Search Results: "vagrant"

6 April 2023

Reproducible Builds: Reproducible Builds in March 2023

Welcome to the March 2023 report from the Reproducible Builds project. In these reports we outline the most important things that we have been up to over the past month. As a quick recap, the motivation behind the reproducible builds effort is to ensure no malicious flaws have been introduced during compilation and distributing processes. It does this by ensuring identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. If you are interested in contributing to the project, please do visit our Contribute page on our website.

News There was progress towards making the Go programming language reproducible this month, with the overall goal remaining making the Go binaries distributed from Google and by Arch Linux (and others) to be bit-for-bit identical. These changes could become part of the upcoming version 1.21 release of Go. An issue in the Go issue tracker (#57120) is being used to follow and record progress on this.
Arnout Engelen updated our website to add and update reproducibility-related links for NixOS to reproducible.nixos.org. [ ]. In addition, Chris Lamb made some cosmetic changes to our presentations and resources page. [ ][ ]
Intel published a guide on how to reproducibly build their Trust Domain Extensions (TDX) firmware. TDX here refers to an Intel technology that combines their existing virtual machine and memory encryption technology with a new kind of virtual machine guest called a Trust Domain. This runs the CPU in a mode that protects the confidentiality of its memory contents and its state from any other software.
A reproducibility-related bug from early 2020 in the GNU GCC compiler as been fixed. The issues was that if GCC was invoked via the `as` frontend, the `-ffile-prefix-map` was being ignored. We were tracking this in Debian via the `build_path_captured_in_assembly_objects` issue. It has now been fixed and will be reflected in GCC version 13.
Holger Levsen will present at foss-north 2023 in April of this year in Gothenburg, Sweden on the topic of Reproducible Builds, the first ten years.
Anthony Andreoli, Anis Lounis, Mourad Debbabi and Aiman Hanna of the Security Research Centre at Concordia University, Montreal published a paper this month entitled On the prevalence of software supply chain attacks: Empirical study and investigative framework:
Software Supply Chain Attacks (SSCAs) typically compromise hosts through trusted but infected software. The intent of this paper is twofold: First, we present an empirical study of the most prominent software supply chain attacks and their characteristics. Second, we propose an investigative framework for identifying, expressing, and evaluating characteristic behaviours of newfound attacks for mitigation and future defense purposes. We hypothesize that these behaviours are statistically malicious, existed in the past, and thus could have been thwarted in modernity through their cementation x-years ago. [ ]

On our mailing list this month:

Mattia Rizzolo is asking everyone in the community to save the date for the 2023 s Reproducible Builds summit which will take place between October 31st and November 2nd at Dock Europe in Hamburg, Germany. Separate announcement(s) to follow. [ ]

ahojlm posted an message announcing a new project which is the first project offering bootstrappable and verifiable builds without any binary seeds. That is to say, a way of providing a verifiable path towards trusted software development platform without relying on pre-provided binary code in order to prevent against various forms of compiler backdoors. The project s homepage is hosted on Tor (mirror).

The minutes and logs from our March 2023 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on Tuesday 25th April at 15:00 UTC on `#reproducible-builds` on the OFTC network.
and as a Valentines Day present, Holger Levsen wrote on his blog on 14th February to express his thanks to OSUOSL for their continuous support of reproducible-builds.org. [ ]

Debian Vagrant Cascadian developed an easier setup for testing debian packages which uses sbuild s unshare mode along and reprotest, our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. [ ]
Over 30 reviews of Debian packages were added, 14 were updated and 7 were removed this month, all adding to our knowledge about identified issues. A number of issues were updated, including the Holger Levsen updating `build_path_captured_in_assembly_objects` to note that it has been fixed for GCC 13 [ ] and Vagrant Cascadian added new issues to mark packages where the build path is being captured via the Rust toolchain [ ] as well as new categorisation for where virtual packages have nondeterministic versioned dependencies [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`cockpit` (gzip mtime)

`crmsh` (by mcepl: rewrite to avoid python toolchain issue)

`cx_Freeze` (merged, FTBFS-2038)

`golangci-lint` (date)

`guestfs-tools` (gzip mtime)

`perf` (merged, sort python scandir)

`perl-Date-Calc-XS` (FTBFS-2038)

`perl-Date-Calc` (FTBFS-2038)

`pw3270` (merged, date)

`python-dtaidistance` (drop unreproducible unnecessary file)

`sonic-pi` (FTBFS-2038)

`spack` (parallelism)

`tesseract` (fixed, CPU, -march=native)

Chris Lamb:

#1032409 filed against `esda`.

#1032759 filed against `gle-graphics-manual`.

Stefan Br ns:

`transfig/fig2dev` (also in openSUSE ; date in PDF)

Vagrant Cascadian:

#1033032 filed against `buddy`.

#1033089 filed against `subread`.

#1033663 filed against `linux`.

In addition, Vagrant Cascadian filed a bug with a patch to ensure GNU Modula-2 supports the `SOURCE_DATE_EPOCH` environment variable.

Testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In March, the following changes were made by Holger Levsen:

Arch Linux-related changes:

Build Arch packages in `/tmp/archlinux-ci/$SRCPACKAGE` instead of `/tmp/$SRCPACKAGE`. [ ]

Start 2/3 of the builds on the `o1` node, the rest on `o2`. [ ]

Add graphs for Arch Linux (and OpenWrt) builds. [ ]

Toggle Arch-related builders to debug why a specific node overloaded. [ ][ ][ ][ ]

Node health checks:

Detect `SetuptoolsDeprecationWarning` tracebacks in Python builds. [ ]

Detect failures do perform `chdist` calls. [ ][ ]

OSUOSL node migration.

Install `megacli` packages that are needed for hardware RAID. [ ][ ]

Add health check and maintenance jobs for new nodes. [ ]

Add mail config for new nodes. [ ][ ]

Handle a node running in the future correctly. [ ][ ]

Migrate some nodes to Debian bookworm. [ ]

Fix nodes health overview for osuosl3. [ ]

Make sure the `/srv/workspace` directory is owned by by the `jenkins` user. [ ]

Use `.debian.net` names everywhere, except when communicating with the outside world. [ ]

Grant fpierret access to a new node. [ ]

Update documentation. [ ]

Misc migration changes. [ ][ ][ ][ ][ ][ ][ ][ ]

Misc changes:

Enable fail2ban everywhere and monitor it with munin [ ].

Gracefully deal with non-existing Alpine schroots. [ ]

In addition, Roland Clobus is continuing his work on reproducible Debian ISO images:

Add/update openQA configuration [ ], and use the actual timestamp for openQA builds [ ].

Moved adding the user to the `docker` group from the `janitor_setup_worker` script to the (more general) `update_jdn.sh` script. [ ]

Use the (short-term) reproducible source when generating `live-build` images. [ ]

diffoscope development diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats as well. This month, Mattia Rizzolo released versions `238`, and Chris Lamb released versions `239` and `240`. Chris Lamb also made the following changes:

Fix compatibility with PyPDF 3.x, and correctly restore test data. [ ]

Rework PDF annotation handling into a separate method. [ ]

In addition, Holger Levsen performed a long-overdue overhaul of the Lintian overrides in the Debian packaging [ ][ ][ ][ ], and Mattia Rizzolo updated the packaging to silence an `include_package_data=True` [ ], fixed the build under Debian bullseye [ ], fixed tool name in a list of tools permitted to be absent during package build tests [ ] and as well as documented sending out an email upon [ ]. In addition, Vagrant Cascadian updated the version of GNU Guix to 238 [ and 239 [ ]. Vagrant also updated reprotest to version 0.7.23. [ ]

Other development work Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

5 March 2023

Reproducible Builds: Reproducible Builds in February 2023

Welcome to the February 2023 report from the Reproducible Builds project. As ever, if you are interested in contributing to our project, please visit the Contribute page on our website.

FOSDEM 2023 was held in Brussels on the 4th & 5th of February and featured a number of talks related to reproducibility. In particular, Akihiro Suda gave a talk titled Bit-for-bit reproducible builds with Dockerfile discussing deterministic timestamps and deterministic apt-get (original announcement). There was also an entire track of talks on Software Bill of Materials (SBOMs). SBOMs are an inventory for software with the intention of increasing the transparency of software components (the US National Telecommunications and Information Administration (NTIA) published a useful Myths vs. Facts document in 2021).
On our mailing list this month, Larry Doolittle was puzzled why the Debian verilator package was not reproducible [ ], but Chris Lamb pointed out that this was due to the use of Python s datetime.fromtimestamp over datetime.utcfromtimestamp [ ].
James Addison also was having issues with a Debian package: in this case, the alembic package. Chris Lamb was also able to identify the Sphinx documentation generator as the cause of the problem, and provided a potential patch that might fix it. This was later filed upstream [ ].
Anthony Harrison wrote to our list twice, first by introducing himself and their background and later to mention the increasing relevance of Software Bill of Materials (SBOMs):

As I am sure everyone is aware, there is a growing interest in [SBOMs] as a way of improving software security and resilience. In the last two years, the US through the Exec Order, the EU through the proposed Cyber Resilience Act (CRA) and this month the UK has issued a consultation paper looking at software security and SBOMs appear very prominently in each publication. [ ]

Tim Retout wrote a blog post discussing AlmaLinux in the context of CentOS, RHEL and supply-chain security in general [ ]:

Alma are generating and publishing Software Bill of Material (SBOM) files for every package; these are becoming a requirement for all software sold to the US federal government. What s more, they are sending these SBOMs to a third party (CodeNotary) who store them in some sort of Merkle tree system to make it difficult for people to tamper with later. This should theoretically allow end users of the distribution to verify the supply chain of the packages they have installed?

Debian

Vagrant Cascadian noted that the Debian bookworm distribution has finally surpassed bullseye for reproducibility: 96.1% vs. 96.0%, despite having over 3500 more packages in the distribution.

Roland Clobus posted his latest update of the status of reproducible Debian ISO images noting that all major desktops build reproducibly with bullseye, bookworm and sid, with the caveat that when non-free firmware is activated, some non-reproducible files are generated .

FC Stegerman submitted a new Intent to Package (ITP) bug report representing an intention to package `repro-apk`, a set of scripts to make Android `.apk` files reproducible.

23 reviews of Debian packages were added, 24 were updated and 20 were removed this month adding to our knowledge about identified issues. A new issue was added and identified by Chris Lamb [ ], and the `timestamps_embedded_in_manpages_by_node_marked_man` issue has been marked as resolved [ ].

F-Droid & Android

This month, F-Droid added 21 apps published with reproducible builds (out of 33 new apps in total), the overview of F-Droid apps published with Reproducible Builds now includes graphs, and there are now also some graphs of F-Droid apps verified by the Verification Server.

FC Stegerman noticed that signatures made by older versions of Android Gradle plugin cannot be copied because the signing method differs too much from that used by apksigner (and signflinger).

FC Stegerman also created a helpful HOWTO page on the F-Droid Wiki detailing how to compare and subsequently make APKs reproducible.

A long-running thread on Hiding data/code in Android APK embedded signatures continued on our mailing list this month; apksigcopier `v1.1.1` and reproducible-apk-tools `v0.2.2` + `v0.2.3` were also announced on the same list.

Lastly, FC Stegerman reported two issues on Google s own issue tracker: one related to a non-deterministic Dependency Info Block [ ] and another about a virtual entry added by the signflinger tool causing unexpected differences between signed and unsigned APKs [ ].

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb released versions `235` and `236`; Mattia Rizzolo later released version `237`. Contributions include:

Chris Lamb:

Fix compatibility with PyPDF2 (re. issue #331) [ ][ ][ ].

Fix compatibility with ImageMagick version 7.1 [ ].

Require at least version 23.1.0 to run the Black source code tests [ ].

Update `debian/tests/control` after merging changes from others [ ].

Don t write test data during a test [ ].

Update copyright years [ ].

Merged a large number of changes from others.

Akihiro Suda edited the `.gitlab-ci.yml` configuration file to ensure that versioned tags are pushed to the container registry [ ].

Daniel Kahn Gillmor provided a way to migrate from PyPDF2 to pypdf (#1029741).

Efraim Flashner updated the tool metadata for `isoinfo` on GNU Guix [ ].

FC Stegerman added support for Android `resources.arsc` files [ ], improved a number of file-matching regular expressions [ ][ ] and added support for Android `dexdump` [ ]; they also fixed a test failure (#1031433) caused by Debian s `black` package having been updated to a newer version.

Mattia Rizzolo:

updated the release documentation [ ],

fixed a number of Flake8 errors [ ][ ],

updated the autopkgtest configuration to only install `aapt` and `dexdump` on architectures where they are available [ ], making sure that the latest diffoscope release is in a good fit for the upcoming Debian bookworm freeze.

reprotest Reprotest version 0.7.23 was uploaded to both PyPI and Debian unstable, including the following changes:

Holger Levsen improved a lot of documentation [ ][ ][ ], tidied the documentation as well [ ][ ], and experimented with a new `--random-locale` flag [ ].

Vagrant Cascadian adjusted reprotest to no longer randomise the build locale and use a UTF-8 supported locale instead [ ] (re. #925879, #1004950), and to also support passing `--vary=locales.locale=LOCALE` to specify the locale to vary [ ].

Separate to this, Vagrant Cascadian started a thread on our mailing list questioning the future development and direction of reprotest.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`aiohttp` (build fails in the future)

`diff-pdf`

`dpdk`

`ebumeter` (CPU-related issue)

`firecracker` (hashmap ordering issue)

`jhead/gcc` (used random temporary directory name)

`libhugetlbfs` (drop unused unreproducible file)

`prosody` (generates nondeterministic example SSL certificates)

`python-sqlalchemy-migrate` (clean files leftover by Sphinx)

`tigervnc` (random RSA key)

Chris Lamb:

#1030708 filed against `gap-browse`.

#1030714 filed against `cwltool`.

#1030715 filed against `adacgi`.

#1030724 filed against `node-marked-man` (forwarded upstream).

#1030727 filed against `multipath-tools`.

#1031030 filed against `ruby-pgplot`.

#1031412 filed against `pysdl2`.

#1031829 filed against `gawk`.

#1032057 filed against `pyproject-api`.

Gioele Barabucci:

#1032056 filed against `systemtap`.

Larry Doolittle:

#1031711 filed against `verilator`.

Vagrant Cascadian:

#1030270 filed against `libreoffice`.

Testing framework The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In February, the following changes were made by Holger Levsen:

Add three new OSUOSL nodes [ ][ ][ ] and decommission the `osuosl174` node [ ].

Change the order of listed Debian architectures to show the 64-bit ones first [ ].

Reduce the frequency that the Debian package sets and `dd-list` HTML pages update [ ].

Sort Tested suite consistently (and Debian unstable first) [ ].

Update the Jenkins shell monitor script to only query disk statistics every 230min [ ] and improve the documentation [ ][ ].

Other development work disorderfs version `0.5.11-3` was uploaded by Holger Levsen, fixing a number of issues with the manual page [ ][ ][ ].
Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.
If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mastodon: @reproducible_builds@fosstodon.org

Mailing list: `rb-general@lists.reproducible-builds.org`

6 February 2023

Reproducible Builds: Reproducible Builds in January 2023

Welcome to the first report for 2023 from the Reproducible Builds project! In these reports we try and outline the most important things that we have been up to over the past month, as well as the most important things in/around the community. As a quick recap, the motivation behind the reproducible builds effort is to ensure no malicious flaws can be deliberately introduced during compilation and distribution of the software that we run on our devices. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

News In a curious turn of events, GitHub first announced this month that the checksums of various Git archives may be subject to change, specifically that because:
the default compression for Git archives has recently changed. As result, archives downloaded from GitHub may have different checksums even though the contents are completely unchanged.
This change (which was brought up on our mailing list last October) would have had quite wide-ranging implications for anyone wishing to validate and verify downloaded archives using cryptographic signatures. However, GitHub reversed this decision, updating their original announcement with a message that We are reverting this change for now. More details to follow. It appears that this was informed in part by an in-depth discussion in the GitHub Community issue tracker.
The Bundesamt f r Sicherheit in der Informationstechnik (BSI) (trans: The Federal Office for Information Security ) is the agency in charge of managing computer and communication security for the German federal government. They recently produced a report that touches on attacks on software supply-chains (Supply-Chain-Angriff). (German PDF)
Contributor Seb35 updated our website to fix broken links to Tails Git repository [ ][ ], and Holger updated a large number of pages around our recent summit in Venice [ ][ ][ ][ ].
Noak J nsson has written an interesting paper entitled The State of Software Diversity in the Software Supply Chain of Ethereum Clients. As the paper outlines:
In this report, the software supply chains of the most popular Ethereum clients are cataloged and analyzed. The dependency graphs of Ethereum clients developed in Go, Rust, and Java, are studied. These client are Geth, Prysm, OpenEthereum, Lighthouse, Besu, and Teku. To do so, their dependency graphs are transformed into a unified format. Quantitative metrics are used to depict the software supply chain of the blockchain. The results show a clear difference in the size of the software supply chain required for the execution layer and consensus layer of Ethereum.

Yongkui Han posted to our mailing list discussing making reproducible builds & GitBOM work together without gitBOM-ID embedding. GitBOM (now renamed to OmniBOR) is a project to enable automatic, verifiable artifact resolution across today s diverse software supply-chains [ ]. In addition, Fabian Keil wrote to us asking whether anyone in the community would be at Chemnitz Linux Days 2023, which is due to take place on 11th and 12th March (event info). Separate to this, Akihiro Suda posted to our mailing list just after the end of the month with a status report of bit-for-bit reproducible Docker/OCI images. As Akihiro mentions in their post, they will be giving a talk at FOSDEM in the Containers devroom titled Bit-for-bit reproducible builds with `Dockerfile` and that my talk will also mention how to pin the apt/dnf/apk/pacman packages with my `repro-get` tool.
The extremely popular Signal messenger app added upstream support for the `SOURCE_DATE_EPOCH` environment variable this month. This means that release tarballs of the Signal desktop client do not embed nondeterministic release information. [ ][ ]

Distribution work

F-Droid & Android There was a very large number of changes in the F-Droid and wider Android ecosystem this month: On January 15th, a blog post entitled Towards a reproducible F-Droid was published on the F-Droid website, outlining the reasons why F-Droid signs published APKs with its own keys and how reproducible builds allow using upstream developers keys instead. In particular:
In response to [ ] criticisms, we started encouraging new apps to enable reproducible builds. It turns out that reproducible builds are not so difficult to achieve for many apps. In the past few months we ve gotten many more reproducible apps in F-Droid than before. Currently we can t highlight which apps are reproducible in the client, so maybe you haven t noticed that there are many new apps signed with upstream developers keys.
(There was a discussion about this post on Hacker News.) In addition:

F-Droid added 13 apps published with reproducible builds this month. [ ]

FC Stegerman outlined a bug where `baseline.profm` files are nondeterministic, developed a workaround, and provided all the details required for a fix. As they note, this issue has now been fixed but the fix is not yet part of an official Android Gradle plugin release.

GitLab user Parwor discovered that the number of CPU cores can affect the reproducibility of `.dex` files. [ ]

FC Stegerman also announced the `0.2.0` and `0.2.1` releases of reproducible-apk-tools, a suite of tools to help make `.apk` files reproducible. Several new subcommands and scripts were added, and a number of bugs were fixed as well [ ][ ]. They also updated the F-Droid website to improve the reproducibility-related documentation. [ ][ ]

On the F-Droid issue tracker, FC Stegerman discussed reproducible builds with one of the developers of the Threema messenger app and reported that Android SDK build-tools `31.0.0` and `32.0.0` (unlike earlier and later versions) have a `zipalign` command that produces incorrect padding.

A number of bugs related to reproducibility were discovered in Android itself. Firstly, the non-deterministic order of `.zip` entries in `.apk` files [ ] and then newline differences between building on Windows versus Linux that can make builds not reproducible as well. [ ] (Note that these links may require a Google account to view.)

And just before the end of the month, FC Stegerman started a thread on our mailing list on the topic of hiding data/code in APK embedded signatures which has been made possible by the Android APK Signature Scheme v2/v3. As part of this, they made an Android app that reads the APK Signing block of its own APK and extracts a payload in order to alter its behaviour called sigblock-code-poc.

Debian As mentioned in last month s report, Vagrant Cascadian has been organising a series of online sprints in order to clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). During January, a sprint took place on the 10th, resulting in the following uploads:

Chris Lamb:

`critcl` (#963600)

`log4cpp` (#1020662)

`logapp` (#1010845)

`nanomsg` (#1001853)

Holger Levsen:

`netkit-rsh` (#1020798)

`wcwidth` (#1005408)

Vagrant Cascadian:

`mc` (#828683)

`gtk-sharp3` (#989965 & #989966)

During this sprint, Holger Levsen filed Debian bug #1028615 to request that the `tracker.debian.org` service display results of reproducible rebuilds, not just reproducible CI results. Elsewhere in Debian, strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, version `1.13.1-1` was uploaded to Debian unstable by Holger Levsen, including a fix by FC Stegerman (obfusk) to update a regular expression for the latest version of `file(1)` [ ]. (#1028892) Lastly, 65 reviews of Debian packages were added, 21 were updated and 35 were removed this month adding to our knowledge about identified issues.

Other distributions In other distributions:

Bernhard M. Wiedemann published another monthly report for reproducibility within openSUSE, as well as a belated report for December 2022.

It was announced that Fedora Rawhide now clamps file modification types to `SOURCE_DATE_EPOCH`. [ ]

Finally, an existing tool called rpmreproduce was (re-)discovered this month, which claims that given a buildinfo file from a RPM package, [it can] generate instructions for attempting to reproduce the binary packages built from the associated source and build information.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions `231`, `232`, `233` and `234` to Debian:

No need for `from future import print_function` import anymore. [ ]

Comment and tidy the `extras_require.json` handling. [ ]

Split inline Python code to generate test `Recommends` into a separate Python script. [ ]

Update `debian/tests/control` after merging support for PyPDF support. [ ]

Correctly catch segfaulting `cd-iccdump` binary. [ ]

Drop some old debugging code. [ ]

Allow ICC tests to (temporarily) fail. [ ]

In addition, FC Stegerman (obfusk) made a number of changes, including:

Updating the `test_text_proper_indentation` test to support the latest version(s) of `file(1)`. [ ]

Use an `extras_require.json` file to store some build/release metadata, instead of accessing the internet. [ ]

Updating an APK-related `file(1)` regular expression. [ ]

On the diffoscope.org website, de-duplicate contributors by e-mail. [ ]

Lastly, Sam James added support for PyPDF version 3 [ ] and Vagrant Cascadian updated a handful of tool references for GNU Guix. [ ][ ]

Upstream patches The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`argparse` (date-related issue)

`asciimatics` (build failure)

`asyncpg` (fails to build in 2032)

`cpython` (fails to build in 2038)

`django` (fails to build in 2038)

`grandorgue` (`.zip`-related issue)

`libarchive` (fails to build in 2038)

`libarchive` (fails to build in 2038)

`librcc` (date)

`mbedtls` (fails to build in 2023)

`mozilla-nss` (fails to build in 2023)

`ocaml-rpm-macros` (fix fallout from an RPM-related change)

`perl HTTP::Cookies` (fails to build in 2038)

`python-aiosmtplib/python-trustme` (fails to build in 2038 due to SSL certificate)

`python-bmap` (fails to build in 2024)

`python-compileall2` (fails to build in 2038)

`taskwarrior` (`python-tasklib` fails to build in 2038)

`taskwarrior` (fix fails to build in 2038)

`wrk` (hash ordering issue)

`xemacs` (fails to build in 2038 stuck)

Chris Lamb:

#1027988 filed against `click`.

#1027992 filed against `towncrier`.

#1028051 filed against `unifrac-tools`.

#1028310 filed against `hamster-time-tracker`.

#1028515 filed against `accel-config`.

#1029295 filed against `python-miio`.

#1029297 filed against `python-graphviz`.

Vagrant Cascadian:

#1029227 filed against `ectrans`.

#1029303 filed against `fiat-ecmwf`.

#1029307 filed against `node-katex`.

#1029800 filed against `aribas`.

#1029801 filed against `tbox`.

#1029807 filed against `borgbackup2`.

#1029809 filed against `dnf-plugins-core`.

#1030057 filed against `refpolicy`.

FC Stegerman:

Several patches for `file(1)` (which is used by reproducible builds tools like diffoscope and strip-nondeterminism) that improve detection of various file formats are now included in the Debian packaging. [ ]

Testing framework The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In January, the following changes were made by Holger Levsen:

Node changes:

Add three new nodes hosted at the Oregon State University Open Source Lab including integrating them into the DNS, maintenance and monitoring systems. [ ][ ][ ][ ][ ]

Debian-related changes:

Only keep diffoscope s HTML output (ie. no `.json` or `.txt`) for LTS suites and older in order to save diskspace on the Jenkins host. [ ]

Re-create `pbuilder` base less frequently for the `stretch`, `bookworm` and `experimental` suites. [ ]

OpenWrt-related changes:

Add gcc-multilib to `OPENWRT_HOST_PACKAGES` and install it on the nodes that need it. [ ]

Detect more problems in the health check when failing to build OpenWrt. [ ]

Misc changes:

Update the `chroot-run` script to correctly manage `/dev` and `/dev/pts`. [ ][ ][ ]

Update the Jenkins shell monitor script to collect disk stats less frequently [ ] and to include various directory stats. [ ][ ]

Update the real year in the configuration in order to be able to detect whether a node is running in the future or not. [ ]

Bump copyright years in the default page footer. [ ]

In addition, Christian Marangi submitted a patch to build OpenWrt packages with the `V=s` flag to enable debugging. [ ]
If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mastodon: @reproducible_builds@fosstodon.org

Mailing list: `rb-general@lists.reproducible-builds.org`

2 February 2023

Valhalla's Things: How To Verify Debian's ARM Installer Images

Thanks to Vagrant on the debian-arm mailing list I ve found that there is a chain of verifiability for the images usually used to install Debian on ARM devices. It s not trivial, so I m writing it down for future reference when I ll need it again.

Download the images from https://ftp.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/ (choose either hd-media or netboot, then SD-card-images and download the firmware.* file for your board as well as partition.img.gz).
Download the checksums file https://ftp.debian.org/debian/dists/bullseye/main/installer-armhf/current/images/SHA256SUMS
Download the Release file from https://ftp.debian.org/debian/dists/bullseye/ ; for convenience the InRelease

Verify the Release file:

gpg --no-default-keyring \
    --keyring /usr/share/keyrings/debian-archive-bullseye-stable.gpg \
    --verify InRelease

Verify the checksums file:
```
awk '/installer-armhf\/current\/images\/SHA256SUMS/  print $1 "
SHA256SUMS" ' InRelease   tail -n 1   sha256sum -c 
```
(I know, I probably can use awk instead of that tail, but it s getting late and I want to publish this).

Verify the actual files, for hd-media:

grep hd-media SHA256SUMS \
  sed 's#hd-media/SD-card-images/##' \
  sha256sum -c \
  grep -v "No such file or directory" \
  grep -v "FAILED open or read" 2> /dev/null

and for netboot:

grep netboot SHA256SUMS \
  sed 's#netboot/SD-card-images/##' \
  sha256sum -c \
  grep -v "No such file or directory" \
  grep -v "FAILED open or read" 2> /dev/null

and check that all of the files you wanted are there with an OK; of course change hd-media with netboot as needed.

And I fully agree that fewer steps would be nice, but this is definitely better than nothing!

20 January 2023

Reproducible Builds (diffoscope): diffoscope 233 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 233. This version includes the following changes:

[ FC Stegerman ]
* Split packaging metadata into an extras_require.json file instead of using
  the pep517 and the pip modules directly. This was causing build failures if
  not using a virtualenv and/or building without internet access.
  (Closes: #1029066, reproducible-builds/diffoscope#325)
[ Vagrant Cascadian ]
* Add an external tool reference for GNU Guix (lzip).
* Drop an external tool reference for GNU Guix (pedump).
[ Chris Lamb ]
* Split inline Python code in shell script to generate test dependencies to a
  separate Python script.
* No need for "from __future__ import print_function" import in setup.py
  anymore.
* Comment and tidy the new extras_require.json handling.

You find out more by visiting the project homepage.

7 January 2023

Reproducible Builds: Reproducible Builds in December 2022

Welcome to the December 2022 report from the Reproducible Builds project.

We are extremely pleased to announce that the dates for the Reproducible Builds Summit in 2023 have been announced in 2022 already:

When: October 31st, November 1st, November 2nd 2023.
Where: Dock Europe, Hamburg, Germany.

We plan to spend three days continuing to the grow of the Reproducible Builds effort. As in previous events, the exact content of the meeting will be shaped by the participants. And, as mentioned in Holger Levsen s post to our mailing list, the dates have been booked and confirmed with the venue, so if you are considering attending, please reserve these dates in your calendar today.

R my Gr nblatt, an associate professor in the T l com Sud-Paris engineering school wrote up his pain points of using Nix and NixOS. Although some of the points do not touch on reproducible builds, R my touches on problems he has encountered with the different kinds of reproducibility that these distributions appear to promise including configuration files affecting the behaviour of systems, the fragility of upstream sources as well as the conventional idea of binary reproducibility.

Morten Linderud reported that he is quietly optimistic that if Go programming language resolves all of its issues with reproducible builds (tracking issue) then the Go binaries distributed from Google and by Arch Linux may be bit-for-bit identical. It s just a bit early to sorta figure out what roadblocks there are. [But] Go bootstraps itself every build, so in theory I think it should be possible.

On December 15th, Holger Levsen published an in-depth interview he performed with David A. Wheeler on supply-chain security and reproducible builds, but it also touches on the biggest challenges in computing as well. This is part of a larger series of posts featuring the projects, companies and individuals who support the Reproducible Builds project. Other instalments include an article featuring the Civil Infrastructure Platform project and followed this up with a post about the Ford Foundation as well as a recent ones about ARDC, the Google Open Source Security Team (GOSST), Jan Nieuwenhuizen on Bootstrappable Builds, GNU Mes and GNU Guix and Hans-Christoph Steiner of the F-Droid project.

A number of changes were made to the Reproducible Builds website and documentation this month, including FC Stegerman adding an F-Droid/apksigcopier example to our embedded signatures page [ ], Holger Levsen making a large number of changes related to the 2022 summit in Venice as well as 2023 s summit in Hamburg [ ][ ][ ][ ] and Simon Butler updated our publications page [ ][ ].

On our mailing list this month, James Addison asked a question about whether there has been any effort to trace the files used by a build system in order to identify the corresponding build-dependency packages. [ ] In addition, Bernhard M. Wiedemann then posed a thought-provoking question asking How to talk to skeptics? , which was occasioned by a colleague who had published a blog post in May 2021 skeptical of reproducible builds. The thread generated a number of replies.

Android news obfusk (FC Stegerman) performed a thought-provoking review of tools designed to determine the difference between two different `.apk` files shipped by a number of free-software instant messenger applications. These scripts are often necessary in the Android/APK ecosystem due to these files containing embedded signatures so the conventional bit-for-bit comparison cannot be used. After detailing a litany of issues with these tools, they come to the conclusion that:
It s quite possible these messengers actually have reproducible builds, but the verification scripts they use don t actually allow us to verify whether they do.
This reflects the consensus view within the Reproducible Builds project: pursuing a situation in language or package ecosystems where binaries are bit-for-bit identical (over requiring a bespoke ecosystem-specific tool) is not a luxury demanded by purist engineers, but rather the only practical way to demonstrate reproducibility. obfusk also announced the first release of their own set of tools on our mailing list. Related to this, obfusk also posted to an issue filed against Mastodon regarding the difficulties of creating bit-by-bit identical APKs, especially with respect to copying v2/v3 APK signatures created by different tools; they also reported that some APK ordering differences were not caused by building on macOS after all, but by using Android Studio [ ] and that F-Droid added 16 more apps published with Reproducible Builds in December.

Debian As mentioned in last months report, Vagrant Cascadian has been organising a series of online sprints in order to clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). During December, meetings were held on the 1st, 8th, 15th, 22nd and 29th, resulting in a large number of uploads and bugs being addressed:

Chris Lamb: `aespipe` (#661079, #1020809), `cdbackup` (#1011428) & `xmlrpc-epi` (#865688, #1020651)

Holger Levsen: `apr-util` (#1006865), `lirc` (#979024) & `ruby-omniauth-tumblr`

Vagrant Cascadian: `amavisd-milter` (#975954), `apophenia` (#940013), `cfi` (#995647), `chessx` (#881664), `cmocka` (#991181), `desmume` (#890312), `golang-gonum-v1-plot` (#968045), `intel-gpu-tools` (#945105), `jhbuild` (#971420), `libjama` (#986601), `libjs-qunit` (#976445), `liblip` (#1001513, #989583), `libstatgrab` (#961747), `mlpost` (#977179 and #977180), `netcdf-parallel` (#972930), `netgen-lvs` (#955783), `perfect-scrollbar` (#1000770), `python-tomli` (#994979), `pytsk` (#992060), `smplayer` (#997689), `squeak-plugins-scratch` (#876771, #942006), `stgit` (#942009), `strace` (#896016), `surgescript` (#992061), `sympow` (#973601), `wxmaxima` (#983148), `xavs2` (#952493), `xaw3d` (#991180, #986704) and `yard` (#972668).

The next sprint is due to take place this coming Tuesday, January 10th at 16:00 UTC.

Upstream patches The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`OpenRGB` (filesystem ordering issue)

`python-maturin` (report an issue regarding random numbers)

`rav1e` (datetime-related issue)

`weblate` (report that the build fails in 2038)

Chris Lamb:

#1025415 filed against `cctools`.

#1025801 filed against `sphinx` (forwarded upstream)

#1026381 filed against `python-django-health-check`.

#1026876 filed against `jamin`.

#1026877 filed against `opari2`.

Testing framework The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:

The `osuosl167` machine is no longer a `openqa-worker` node anymore. [ ][ ]

Detect problems with APT repository signatures [ ] and update a repository signing key [ ].

reproducible Debian builtin-pho: improve job output. [ ]

Only install the `foot-terminfo` package on Debian systems. [ ]

In addition, Mattia Rizzolo added support for the version of diffoscope in Debian stretch which doesn t support the `--timeout` flag. [ ][ ]

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions `228`, `229` and `230` to Debian:

Fix compatibility with `file(1)` version 5.43, with thanks to Christoph Biedl. [ ]

Skip the `test_html.py::test_diff` test if `html2text` is not installed. (#1026034)

Update copyright years. [ ]

In addition, Jelle van der Waa added support for Berkeley DB version 6. [ ] Orthogonal to this, Holger Levsen bumped the Debian `Standards-Version` on all of our packages, including diffoscope [ ], strip-nondeterminism [ ], disorderfs [ ] and reprotest [ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. You can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

8 December 2022

Reproducible Builds: Reproducible Builds in November 2022

Welcome to yet another report from the Reproducible Builds project, this time for November 2022. In all of these reports (which we have been publishing regularly since May 2015) we attempt to outline the most important things that we have been up to over the past month. As always, if you interested in contributing to the project, please visit our Contribute page on our website.

Reproducible Builds Summit 2022 Following-up from last month s report about our recent summit in Venice, Italy, a comprehensive report from the meeting has not been finalised yet watch this space! As a very small preview, however, we can link to several issues that were filed about the website during the summit (#38, #39, #40, #41, #42, #43, etc.) and collectively learned about Software Bill of Materials (SBOM) s and how `.buildinfo` files can be seen/used as SBOMs. And, no less importantly, the Reproducible Builds t-shirt design has been updated

Reproducible Builds at European Cyber Week 2022 During the European Cyber Week 2022, a Capture The Flag (CTF) cybersecurity challenge was created by Fr d ric Pierret on the subject of Reproducible Builds. The challenge consisted in a pedagogical sense based on how to make a software release reproducible. To progress through the challenge issues that affect the reproducibility of build (such as build path, timestamps, file ordering, etc.) were to be fixed in steps in order to get the final flag in order to win the challenge. At the end of the competition, five people succeeded in solving the challenge, all of whom were awarded with a shirt. Fr d ric Pierret intends to create similar challenge in the form of a how to in the Reproducible Builds documentation, but two of the 2022 winners are shown here:

On business adoption and use of reproducible builds Simon Butler announced on the rb-general mailing list that the Software Quality Journal published an article called On business adoption and use of reproducible builds for open and closed source software. This article is an interview-based study which focuses on the adoption and uses of Reproducible Builds in industry, with a focus on investigating the reasons why organisations might not have adopted them:
[ ] industry application of R-Bs appears limited, and we seek to understand whether awareness is low or if significant technical and business reasons prevent wider adoption.
This is achieved through interviews with software practitioners and business managers, and touches on both the business and technical reasons supporting the adoption (or not) of Reproducible Builds. The article also begins with an excellent explanation and literature review, and even introduces a new helpful analogy for reproducible builds:
[Users are] able to perform a bitwise comparison of the two binaries to verify that they are identical and that the distributed binary is indeed built from the source code in the way the provider claims. Applied in this manner, R-Bs function as a canary, a mechanism that indicates when something might be wrong, and offer an improvement in security over running unverified binaries on computer systems.
The full paper is available to download on an open access basis. Elsewhere in academia, Beatriz Michelson Reichert and Rafael R. Obelheiro have published a paper proposing a systematic threat model for a generic software development pipeline identifying possible mitigations for each threat (PDF). Under the Tampering rubric of their paper, various attacks against Continuous Integration (CI) processes:
An attacker may insert a backdoor into a CI or build tool and thus introduce vulnerabilities into the software (resulting in an improper build). To avoid this threat, it is the developer s responsibility to take due care when making use of third-party build tools. Tampered compilers can be mitigated using diversity, as in the diverse double compiling (DDC) technique. Reproducible builds, a recent research topic, can also provide mitigation for this problem. (PDF)

Misc news

A change was proposed for the Go programming language to enable reproducible builds when Link Time Optimisation (LTO) is enabled. As mentioned in the changelog, Morten Linderud s patch fixes two issues when the linker used in conjunction with the `-flto` option: the first involves solving an issue related to seeded random numbers; and the second involved the binary embedding the current working directory in compressed sections of the LTO object. Both of these issues made the build unreproducible.

In the .NET framework ecosystem, a wiki page for the Roslyn .NET C# and Visual Basic compiler was uncovered this month that details its attempts to ensure end-to-end reproducible builds by focusing on the definition on what are considered inputs to the compiler for the purpose of determinism . This is a spiritual followup to a 2016 blog post by Microsoft developer Jared Parsons on Deterministic builds in Roslyn which starts: It seems silly to celebrate features which should have been there from the start.

Ian Lance Taylor followed up an old post to report that Jakub Jelinek s patch from September 2000 is incomplete.

In F-Droid this month, Reproducible Builds contributor FC Stegerman created a set of reproducible APK tools as a workaround for issues like the order of files in APKs built on macOS being non-deterministic. In addition, the new issue documenting the overview of apps using reproducible builds shows that F-Droid added 11 new apps that use reproducible builds, and FC Stegerman released apksigcopier version 1.1.0 which adds support for APKs signed by Signflinger .

martinSusz has written up a fascinating wiki page describing how to generate quasi-reproducible firmware ROMs for System-on-a-Chip (SoC) components fabricated by Rock Chip. These chips are used in popular low-cost laptops such as the Pine64 PinebookPro and Asus C201. The link is worth viewing simply for the interesting diagram.

Our monthly IRC meeting was held on November 29th 2022. Our next meeting will be on January 31st 2023; we ll skip the meeting in December due to the proximity to Christmas, etc.

On our mailing list this month:

Adrian Diglio from Microsoft asked How to Add a New Project within Reproducible Builds which solicited a number of replies.

Vagrant Cascadian posed an interesting question regarding the difference between test builds vs rebuilds (or verification rebuilds ). As Vagrant poses in their message, they re both useful for slightly different purposes, and it might be good to clarify the distinction [ ].

Debian & other Linux distributions Over 50 reviews of Debian packages were added this month, another 48 were updated and almost 30 were removed, all of which adds to our knowledge about identified issues. Two new issue types were added as well. [ ][ ]. Vagrant Cascadian announced on our mailing list another online sprint to help clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but others were held on October 6th and October 20th. There were two additional sprints that occurred in November, however, which resulted in the following progress:

Chris Lamb:

`paxctl` (Fixed #1020804)

`png23d` (Fixed #1020805)

`tuxcmd-modules` (Fixed #1011500 & #941296)

`waili` (Fixed #1020751)

`zephyr` (Fixed #828867 #1021374)

Vagrant Cascadian:

`ddd` (Fixed #834016)

`libpam-ldap` (Fixed #834050)

`nsnake` (Fixed #833612)

`quvi` (Fixed #835259)

`stressapptest` (Fixed #831587 & #986653)

`tcpreen` (Fixed #831585)

`boolector` (Fixed #1023886)

`tsdecrypt` (Fixed #829713 & #1022130)

`wbxml2` (QA upload fixed build path issues)

`tercpp` (QA upload fixed build path issues)

Lastly, Roland Clobus posted his latest update of the status of reproducible Debian ISO images on our mailing list. This reports that all major desktops build reproducibly with bullseye, bookworm and sid as well as that no custom patches needed to applied to Debian unstable for this result to occur. During November, however, Roland proposed some modifications to live-setup and the rebuild script has been adjusted to fix the failing Jenkins tests for Debian bullseye [ ][ ].
In other news, Miro Hron ok proposed a change to clamp build modification times to the value of `SOURCE_DATE_EPOCH`. This was initially suggested and discussed on a `devel@` mailing list post but was later written up on the Fedora Wiki as well as being officially proposed to Fedora Engineering Steering Committee (FESCo).

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`dwz` (Profile-guided optimisation issue)

`icmake` (filesystem ordering issue)

`llmnrd`

`elixir` (report a bug re. stuck build on single-core VMs)

`warzone2100` (report a bug re. parallelism-dependent output)

Chris Lamb:

#1023589 filed against `libnvme`.

#1024352 filed against `pykafka`.

Vagrant Cascadian:

#1023886 filed against `boolector`.

#1023956 filed against `fl-cow`.

#1023957 filed against `gerstensaft`.

#1023960 filed against `libcgicc`.

#1024007 filed against `haskell98-report`.

#1024125 filed against `ucspi-proxy`.

#1024126 filed against `hunt`.

#1024279 filed against `tolua++`.

#1024282 filed against `twoftpd`.

#1024283 filed against `ipsvd`.

#1024284 filed against `gentoo`.

#1024286 filed against `lcm`.

#1024288 filed against `apcupsd`.

#1024289 filed against `openfortivpn`.

#1024290 filed against `xtb`.

#1024291 filed against `gnunet`.

#1024292 filed against `swift-im`.

#1024396 filed against `brewtarget`.

#1024399 filed against `xrprof`.

#1024404 filed against `gitlint`.

#1024412 filed against `claws-mail`.

#1024413 filed against `presage`.

#1024530 filed against `jh7100-bootloader-recovery`.

Victor Westerhuis:

#1024482 & #1024638 filed against `opencv`.

John Neffenger:

`tomcat` (Fixed Apache bug #66346)

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `226` and `227` to Debian:

Support both `python3-progressbar` and `python3-progressbar2`, two modules providing the `progressbar` Python module. [ ]

Don t run Python decompiling tests on Python bytecode that `file(1)` cannot detect yet and Python 3.11 cannot unmarshal. (#1024335)

Don t attempt to attach text-only differences notice if there are no differences to begin with. (#1024171)

Make sure we recommend `apksigcopier`. [ ]

Tidy generation of `os_list`. [ ]

Make the code clearer around generating the Debian substvars . [ ]

Use our `assert_diff` helper in `test_lzip.py`. [ ]

Drop other copyright notices from `lzip.py` and `test_lzip.py`. [ ]

In addition to this, Christopher Baines added lzip support [ ], and FC Stegerman added an optimisation whereby we don t run `apktool` if no differences are detected before the signing block [ ].
A significant number of changes were made to the Reproducible Builds website and documentation this month, including Chris Lamb ensuring the openEuler logo is correctly visible with a white background [ ], FC Stegerman de-duplicated by email address to avoid listing some contributors twice [ ], Herv Boutemy added Apache Maven to the list of affiliated projects [ ] and boyska updated our Contribute page to remark that the Reproducible Builds presence on salsa.debian.org is not just the Git repository but is also for creating issues [ ][ ]. In addition to all this, however, Holger Levsen made the following changes:

Add a number of existing publications [ ][ ] and update metadata for some existing publications as well [ ].

Hide draft posts on the website homepage. [ ]

Add the Warpforge build tool as a participating project of the summit. [ ]

Clarify in the footer that we welcome patches to the website repository. [ ]

Testing framework The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:

Improve the generation of meta package sets (used in grouping packages for reporting/statistical purposes) to treat Debian bookworm as equivalent to Debian unstable in this specific case [ ] and to parse the list of packages used in the Debian cloud images [ ][ ][ ].

Temporarily allow Frederic to `ssh(1)` into our snapshot server as the `jenkins` user. [ ]

Keep some reproducible jobs Jenkins logs much longer [ ] (later reverted).

Improve the node health checks to detect failures to update the Debian cloud image package set [ ][ ] and to improve prioritisation of some kernel warnings [ ].

Always echo any IRC output to Jenkins output as well. [ ]

Deal gracefully with problems related to processing the cloud image package set. [ ]

Finally, Roland Clobus continued his work on testing Live Debian images, including adding support for specifying the origin of the Debian installer [ ] and to warn when the image has unmet dependencies in the package list (e.g. due to a transition) [ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. You can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

29 November 2022

Sam Hartman: Introducing Carthage

For the past four years, I ve been working on Carthage, a free-software Infrastructure as Code framework. We ve finally reached a point where it makes sense to talk about Carthage and what it can do. This is the first in a series of blog posts to introduce Carthage, discuss what it can do and show how it works. Why Another IAC Framework It seems everywhere you look, there are products designed to support the IAC pattern. On the simple side, you could check a Containerfile into Git. Products like Terraform and Vagrant allow you to template cloud infrastructure and VMs. There are more commercial offerings than I can keep up with. We were disappointed by what was out there when we started Carthage. Other products have improved, but for many of our applications we re happy with what Carthage can build. The biggest challenge we ran into is that products wanted us to specify things at the wrong level. For some of our cyber training work we wanted to say things like We want 3 blue teams, each with a couple defended networks, a red team, and some neutral infrastructure for red to exploit. Yet the tools we were trying to use wanted to lay things out at the individual machine/container level. We found ourselves contemplating writing a program to generate input for some other IAC tool. Things were worse for our internal testing. Sometimes we d be shipping hardware to a customer. But sometimes we d be virtualizing that build out in a lab. Sometimes we d be doing a mixture. So we wanted to completely separate the descriptions of machines, networks, and software from any of the information about whether that was realized on hardware, VMs, containers, or a mixture. Dimensional Breakdown In discussing Carthage with Enrico Zini, he pointed me at Cognitive Dimensions of notation as a way to think about how Carthage approaches the IAC problem. I m more interested in the idea of breaking down a design along the idea of dimensions that allow examining the design space than I am particular adherence to Green s original dimensions. Low Viscosity, High Abstraction Reuse One of the guiding principles is that we want to be able to reuse different components at different scales and in different environments. These include being able to do things like:

Define an operation like Update a Debian system and apply that in several environments including as part of building a base VM or container image, applying to an independently managed machine, or applying to a micro service container that does not run services like ssh or systemd.
Defining a role like DNS server that can be applied to a dedicated machine only having that role, to a traditional server with multiple roles, or in a micro service environment.
Allowing people to write groups of functionality that can be useful in descriptions of a small number of machines, but can also be reused in large environments like modeling of cyber infrastructure to defend. In the small environments, things are simplified, but in larger environments integration like directories, authentication infrastructure and the like is needed.
Allow grouping of functionality at multiple levels. So far I have talked about grouping of software to be installed on a single machine or container. We also want to allow groups of containers (pods or otherwise), groups of machines, groups of networks, or even enclaves (think a model of an entire company or section of a company). Each kind of grouping needs to be parametric and reusable.

Hidden Dependencies To accomplish these abstraction goals, dependencies need to be non-local. For example, a software role might need to integrate with a directory if a directory is present in the environment. When writing the role, no one is going to know which directory to use, nor whether a directory is present. Taking that as an explicit input into the role is error-prone when the role is combined into large abstract units (bigger roles or collections of machines). Instead it is better to have a non-local dependency, and to find the directory if it is available. We accomplish this using dependency injection. In addition to being non-local, dependencies are sometimes hidden. It is very easy to overwhelm our cognitive capacity with even a fairly simple IAC description. An effective notation allows us to focus on the parts that matter when working with a particular part of the description. I ve found hiding dependencies, especially indirect dependencies, to be essential in building complex descriptions. Obviously, tools are required for examining these dependencies as part of debugging. First Class Modeling Clearly one of the goals of IAC descriptions is to actually build and manage infrastructure. It turns out that there are all sorts of things you want to do with the description well before you instantiate the infrastructure. You might want to query the description to build network diagrams, understand interdependencies, or even build inventory/bill of materials. We often find ourselves building Ansible inventory, switch configurations, DNS zones, and all sorts of configuration artifacts. These artifacts may be installed into infrastructure that is instantiated by the description, but they may be consumed in other ways. Allowing the artifacts to be consumed externally means that you can avoid pre-commitment and focus on whatever part of the description you originally want to work on. You may use an existing network at first. Later the IAC description may replace that, or perhaps it never will. As a result, Carthage separates modeling from instantiation. The model can generally be built and queried without needing to interact with clouds, VMs, or containers. We ve actually found it useful to build Carthage layouts that cannot ever be fully instantiated, for example because they never specify details like whether a model should be instantiated on a container or VM, or what kind of technology will realize a modeled network. This allows developing roles before the machines that use them or focusing on how machines will interact and how the network will be laid out before the details of installing on specific hardware. The modeling separation is by far the difference I value most between Carthage and other systems. A Tool for Experts. In Neal Stephenson s essay In the Beginning Was the Command Line , Stephenson points out that the kind of tools experts need are not the same tools that beginners need. The illustration of why a beginner might not be satisfied with a Hole Hog drill caught my attention. Carthage is a tool for experts. Despite what cloud providers will tell you, IAC is not easy. Doubly so when you start making reusable components. Trying to hide that or focus on making things easy to get started can make it harder for experts to efficiently solve the problems they are facing. When we have faced trade offs between making Carthage easy to pick up and making it powerful for expert users, we have chosen to support the experts. That said, Carthage today is harder to pick up than it needs to be. It s a relatively new project with few external users as of this time. Our documentation and examples need improvement, just like every project at this level of maturity. Similarly, as the set of things people try to do expand, we will doubtless run into bugs that our current test cases don t cover. So Carthage absolutely will get easier to learn and use than it is today. Also, we ve already had success building beginner-focused applications on top of Carthage. For our cyber training, we built web applications on top of Carthage that made rebuilding and exploring infrastructure easy. We ve had success using relatively understood tools like Ansible as integration and customization points for Carthage layouts. But in all these cases, when the core layout had significant reusable components and significant complexity in the networking, only an IAC expert was going to be able to maintain and develop that layout. What Carthage can do. Carthage has a number of capabilities today. One of Carthage s strengths is its extensible design. Abstract interfaces make it easy to add new virtualization platforms, cloud services, and support for various ways of managing real hardware. This approach has been validated by incrementally adding support for virtualization architectures and cloud services. As development has progressed, adding new integrations continues to get faster because we are able to reuse existing infrastructure. Today, Carthage can model:

Machines
Networks
Dynamically compose groupings of the above
Generate model level artifacts
- Ansible inventory
- Various DNS integrations
- Various switch configurations

Carthage has excellent facilities for dealing with images on which VMs and Containers can be based, although it does have a bit of a Debian/Ubuntu bias in how it thinks about images:

Building base images from a tool like debootstrap
Customizing these images
Converting into VM images for kvm, VMware, and AWS
Building from scratch OCI images for Podman, Docker and k8s
Adding layers to existing OCI images

When instantiating infrastructure, Carthage can work with:

systemd nspawn containers
Podman (Docker would be easy)
Libvirt
VMware
With the AWS plugin, EC2 VMs and networking

We have also looked at Oracle Cloud and I believe Openstack, although that code is not merged. Future posts will talk about core Carthage concepts and how to use Carthage to build infrastructure.

comments

11 November 2022

Reproducible Builds: Reproducible Builds in October 2022

Welcome to the Reproducible Builds report for October 2022! In these reports we attempt to outline the most important things that we have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Our in-person summit this year was held in the past few days in Venice, Italy. Activity and news from the summit will therefore be covered in next month s report!

A new article related to reproducible builds was recently published in the 2023 IEEE Symposium on Security and Privacy. Titled Taxonomy of Attacks on Open-Source Software Supply Chains and authored by Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais, their paper:

[ ] proposes a general taxonomy for attacks on opensource supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution.

Taking the form of an attack tree, the paper covers 107 unique vectors linked to 94 real world supply-chain incidents which is then mapped to 33 mitigating safeguards including, of course, reproducible builds:

Reproducible Builds received a very high utility rating (5) from 10 participants (58.8%), but also a high-cost rating (4 or 5) from 12 (70.6%). One expert commented that a reproducible build like used by Solarwinds now, is a good measure against tampering with a single build system and another claimed this is going to be the single, biggest barrier .

It was noticed this month that Solarwinds published a whitepaper back in December 2021 in order to:

[ ] illustrate a concerning new reality for the software industry and illuminates the increasingly sophisticated threats made by outside nation-states to the supply chains and infrastructure on which we all rely.

The 12-month anniversary of the 2020 Solarwinds attack (which SolarWinds Worldwide LLC itself calls the SUNBURST attack) was, of course, the likely impetus for publication.

Whilst collaborating on making the Cyrus IMAP server reproducible, Ellie Timoney asked why the Reproducible Builds testing framework uses two remarkably distinctive build paths when attempting to flush out builds that vary on the absolute system path in which they were built. In the case of the Cyrus IMAP server, these happened to be:

/build/1st/cyrus-imapd-3.6.0~beta3/
/build/2/cyrus-imapd-3.6.0~beta3/2nd/

Asked why they vary in three different ways, Chris Lamb listed in detail the motivation behind to each difference.

On our mailing list this month:

Daniel Garcia from WalletScrutiny.com started a thread asking for input on buttons with the Reproducible Builds logo, requesting design suggestions or other feedback. [ ]
Arch Linux contributor kpcyrd wrote to our list this month with the news that multiple people in Arch Linux noticed the output of our git archive command doesn t match the tarball served by GitHub anymore. In his post, kpcyrd narrows the change to a specific commit in Git. [ ]
Akihiro Suda wrote to a share a new tool called repro-get. According to Akihiro s post, repro-get is a tool to install a specific snapshot of apt/dnf/apk/pacman packages using SHA256SUMS files . This is needed in order to install specific (or pinned ) dependencies needed to validate a build.
Finally, Janneke Nieuwenhuizen announced the release of GNU Mes 0.24.1, which represents 23 commits over five months by four people. GNU Mes is a Scheme interpreter and C compiler for bootstrapping the GNU System. [ ]

The Reproducible Builds project is delighted to welcome openEuler to the Involved projects page [ ]. openEuler is Linux distribution developed by Huawei, a counterpart to it s more commercially-oriented EulerOS.

Debian Colin Watson wrote about his experience towards making the databases generated by the `man-db` UNIX manual page indexing tool:
One of the people working on [reproducible builds] noticed that man-db s database files were an obstacle to [reproducibility]: in particular, the exact contents of the database seemed to depend on the order in which files were scanned when building it. The reporter proposed solving this by processing files in sorted order, but I wasn t keen on that approach: firstly because it would mean we could no longer process files in an order that makes it more efficient to read them all from disk (still valuable on rotational disks), but mostly because the differences seemed to point to other bugs.
Colin goes on to describe his approach to solving the problem, including fixing various fits of internal caching, and he ends his post with None of this is particularly glamorous work, but it paid off .
Vagrant Cascadian announced on our mailing list another online sprint to help clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). The first such sprint took place on September 22nd, but another was held on October 6th, and another small one on October 20th. This resulted in the following progress:

Chris Lamb:

`ascii2binary` (Fixed #1020812, #998758 & #1007421)

`bibclean` (Fixed #829754 & #929036)

`dradio` (Fixed #1020814)

`leave` (Fixed #777403, #967002 & #999259)

`libimage-imlib2-perl` (Fixed #1020665)

`mailto` (Fixed #998978 & #777413)

`remote-tty` (Fixed #829721 & #977280)

`xcolmix` (Fixed #1020748, #999219 & #988018)

`z80asm` (Fixed #939775 & #1020875)

Holger Levsen:

`libtheora` (Fixed #990843 & #990844)

`sgml-base` (Fixed #1006646 & #929706)

Vagrant Cascadian:

`ario` (Investigated #828876)

`cloop` (Fixed #787996)

`elvis-tiny` (Fixed #829755 & #901345)

`hannah` (Fixed #845782 & #901260)

`mc` (Investigated #828683)

`mod-dnssd` (Submitted alternate fix for #828752)

`snake4` (Fixed #829715 & #913734)

`the` (Fixed #842550)

`zephyr` (Investigated #828867 & #1021374)

`msp430mcu` (Fixed #860275)

`checkpw` (Fixed #777299 & #1020887)

`madlib` (Fixed #778946)

41 reviews of Debian packages were added, 62 were updated and 12 were removed this month adding to our knowledge about identified issues. A number of issue types were updated too. [1][ ]
Lastly, Luca Boccassi submitted a patch to `debhelper`, a set of tools used in the packaging of the majority of Debian packages. The patch addressed an issue in the `dh_installsysusers` utility so that the `postinst` post-installation script that `debhelper` generates the same data regardless of the underlying filesystem ordering.

Other distributions F-Droid is a community-run app store that provides free software applications for Android phones. This month, F-Droid changed their documentation and guidance to now explicitly encourage RB for new apps [ ][ ], and FC Stegerman created an extremely in-depth issue on GitLab concerning the APK signing block. You can read more about F-Droid s approach to reproducibility in our July 2022 interview with Hans-Christoph Steiner of the F-Droid Project. In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`asymptote` (date-related issue)

`fastjet-contrib` (sort nondeterminstic filesystem ordering)

`forge` (Sphinx doctree issue)

`gau2grid` (output varies with `march=native`)

`gosec` (date-related issue)

`helmfile` (date-related issue)

`libnvme` (date-related issue)

`moab` (CPU)

`tcl` (fails to build in 2038)

`vectorscan` (output varies with `march=native`)

`xz2/lzma` (Rust-related filesystem ordering)

Chris Lamb:

#891263 filed against `puppet` back in early 2018 was finally merged into Puppet and was released in Puppet 7.20.0.

#1021198 filed against `puppet-agent`.

#1022777 filed against `tpm2-pytss` (forwarded upstream).

Vagrant Cascadian:

#1021331 filed against `cclive`.

#1021373 filed against `librep`.

#1021374 filed against `zephyr`.

#1021452 filed against `libdv`.

#1021454 filed against `dbview`.

#1021456 filed against `bwbasic`.

#1021457 filed against `olpc-powerd`.

#1021458 filed against `o3dgc`.

#1021461 filed against `icon`.

#1021463 filed against `rdist`.

#1021464 filed against `stfl`.

#1021466 filed against `pacman`.

#1021469 filed against `lam`.

#1021470 filed against `xsok`.

#1021471 filed against `python-djvulibre`.

#1021472 filed against `xzoom`.

#1021473 filed against `nitpic`.

#1021498 filed against `tcm`.

#1021509 filed against `xxkb`.

#1021512 filed against `yersinia`.

#1021513 filed against `centrifuge`.

#1021514 and #1021516 filed against `ssocr`.

#1021518 filed against `jakarta-jmeter`.

#1021520 filed against `guymager`.

#1021521 and #1021522 filed against `crack`.

#1021751 filed against `dc3dd`.

#1021789 filed against `dlt-viewer`.

#1021792 and #1021793 filed against `vart`.

#1021799 and #1021800 filed against `pgrouting`.

#1021860 filed against `libsx`.

#1021893 filed against `device-tree-compiler`.

#1022130 filed against `tsdecrypt`.

John Neffenger:

`openjdk` (Fixed JDK-8292892)

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `224` and `225` to Debian:

Add support for comparing the text content of HTML files using `html2text`. [ ]

Add support for detecting ordering-only differences in XML files. [ ]

Fix an issue with detecting ordering differences. [ ]

Use the capitalised version of Ordering consistently everywhere in output. [ ]

Add support for displaying font metadata using `ttx(1)` from the fonttools suite. [ ]

Testsuite improvements:

Temporarily allow the `stable-po` pipeline to fail in the CI. [ ]

Rename the `order1.diff` test fixture to `json_expected_ordering_diff`. [ ]

Tidy the JSON tests. [ ]

Use `assert_diff` over `get_data` and an manual assert within the XML tests. [ ]

Drop the `ALLOWED_TEST_FILES` test; it was mostly just annoying. [ ]

Tidy the `tests/test_source.py` file. [ ]

Chris Lamb also added a link to diffoscope s OpenBSD packaging on the diffoscope.org homepage [ ] and Mattia Rizzolo fix an test failure that was occurring under with LLVM 15 [ ].

Testing framework The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In October, the following changes were made by Holger Levsen:

Run the `logparse` tool to analyse results on the Debian Edu build logs. [ ]

Install `btop(1)` on all nodes running Debian. [ ]

Switch Arch Linux from using SHA1 to SHA256. [ ]

When checking Debian `debstrap` jobs, correctly log the tool usage. [ ]

Cleanup more task-related temporary directory names when testing Debian packages. [ ][ ]

Use the `cdebootstrap-static` binary for the 2nd runs of the `cdebootstrap` tests. [ ]

Drop a workaround when testing OpenWrt and coreboot as the issue in diffoscope has now been fixed. [ ]

Turn on an `rm(1)` warning into an info -level message. [ ]

Special case the `osuosl168` node for running Debian bookworm already. [ ][ ]

Use the new `non-free-firmware` suite on the `o168` node. [ ]

In addition, Mattia Rizzolo made the following changes:

Ensure that 2nd build has a merged `/usr`. [ ]

Only reconfigure the `usrmerge` package on Debian bookworm and above. [ ]

Fix `bc(1)` syntax in the computation of the percentage of unreproducible packages in the dashboard. [ ][ ][ ]

In the `index_suite_` pages, order the package status to be the same order of the menu. [ ]

Pass the `--distribution` parameter to the `pbuilder` utility. [ ]

Finally, Roland Clobus continued his work on testing Live Debian images. In particular, he extended the maintenance script to warn when workspace directories cannot be deleted. [ ]
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

16 October 2022

Sven Hoexter: CentOS 9, stunnel, an openssl memory leak and a VirtualBox crash

tl;dr; OpenSSL 3.0.1 leaks memory in ssl3_setup_write_buffer(), seems to be fixed in ~~3.0.5~~ 3.0.2. The issue manifests at least in stunnel and keepalived on CentOS 9. In addition I learned the hard way that running a not so recent VirtualBox version on Debian bullseye let to dh parameter generation crashing in libcrypto in bn_sqr8x_internal(). A recent rabbit hole I went down. The actual bug in openssl was nailed down and documented by Quentin Armitage on GitHub in keepalived My bugreport with all back and forth in the RedHat Bugzilla is #2128412. Act I - Hello stunnel, this is the OOMkiller Calling We started to use stunnel on Google Cloud compute engine instances running CentOS 9. The loadbalancer in front of those instances used a TCP health check to validate the backend availability. A day or so later the stunnel instances got killed by the OOMkiller. Restarting stunnel and looking into /proc/<pid>/smaps showed a heap segment growing quite quickly. Act II - Reproducing the Issue While I'm not the biggest fan of VirtualBox and Vagrant I've to admit it's quite nice to just fire up a VM image, and give other people a chance to recreate that setup as well. Since VirtualBox is no longer released with Debian/stable I just recompiled what was available in unstable at the time of the bullseye release, and used that. That enabled me now to just start a CentOS 9 VM, setup stunnel with a minimal config, grab netcat and a for loop and watch the memory grow. E.g. while true; do nc -z localhost 2600; sleep 1; done To my surprise, in addition to the memory leak, I also observed some crashes but did not yet care too much about those. Act III - Wrong Suspect, a Workaround and Bugreporting Of course the first idea was that something must be wrong in stunnel itself. But I could not find any recent bugreports. My assumption is that there are still a few people around using CentOS and stunnel, so someone else should probably have seen it before. Just to be sure I recompiled the latest stunnel package from Fedora. Didn't change anything. Next I recompiled it without almost all the patches Fedora/RedHat carries. Nope, no progress. Next idea: Maybe this is related to the fact that we do not initiate a TLS context after connecting? So we changed the test case from nc to openssl s_client, and the loadbalancer healthcheck from TCP to a TLS based one. Tada, a workaround, no more memory leaking. In addition I gave Fedora a try (they have Vagrant Virtualbox images in the "Cloud" Spin, e.g. here for Fedora 36) and my local Debian installation a try. No leaks experienced on both. Next I reported #2128412. Act IV - Crash in libcrypto and a VirtualBox Bug When I moved with the test case from the Google Cloud compute instance to my local VM I encountered some crashes. That morphed into a real problem when I started to run stunnel with gdb and valgrind. All crashes happened in libcrypto bn_sqr8x_internal() when generating new dh parameter (stunnel does that for you if you do not use static dh parameter). I quickly worked around that by generating static dh parameter for stunnel. After some back and forth I suspected VirtualBox as the culprit. Recompiling the current VirtualBox version (6.1.38-dfsg-3) from unstable on bullseye works without any changes. Upgrading actually fixed that issue. Epilog I highly appreciate that RedHat, with all the bashing around the future of CentOS, still works on community contributed bugreports. My kudos go to Clemens Lang.

Now that the root cause is clear, I guess RedHat will push out a fix for the openssl 3.0.1 based release they have in RHEL/CentOS 9. Until that is available at least stunnel and keepalived are known to be affected. If you run stunnel on something public it's not that pretty, because already a low rate of TCP connections will result in a DoS condition.

7 October 2022

Reproducible Builds: Reproducible Builds in September 2022

Welcome to the September 2022 report from the Reproducible Builds project! In our reports we try to outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. If you are interested in contributing to the project, please visit our Contribute page on our website.

David A. Wheeler reported to us that the US National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and the Office of the Director of National Intelligence (ODNI) have released a document called Securing the Software Supply Chain: Recommended Practices Guide for Developers (PDF). As David remarked in his post to our mailing list, it expressly recommends having reproducible builds as part of advanced recommended mitigations . The publication of this document has been accompanied by a press release.

Holger Levsen was made aware of a small Microsoft project called oss-reproducible. Part of, OSSGadget, a larger collection of tools for analyzing open source packages , the purpose of oss-reproducible is to:

analyze open source packages for reproducibility. We start with an existing package (for example, the NPM left-pad package, version 1.3.0), and we try to answer the question, Do the package contents authentically reflect the purported source code?

More details can be found in the README.md file within the code repository.

David A. Wheeler also pointed out that there are some potential upcoming changes to the OpenSSF Best Practices badge for open source software in relation to reproducibility. Whilst the badge programme has three certification levels ( passing , silver and gold ), the gold level includes the criterion that The project MUST have a reproducible build . David reported that some projects have argued that this reproducibility criterion should be slightly relaxed as outlined in an issue on the best-practices-badge GitHub project. Essentially, though, the claim is that the reproducibility requirement doesn t make sense for projects that do not release built software, and that timestamp differences by themselves don t necessarily indicate malicious changes. Numerous pragmatic problems around excluding timestamps were raised in the discussion of the issue.

Sonatype, a pioneer of software supply chain management , issued a press release month to report that they had found:

[ ] a massive year-over-year increase in cyberattacks aimed at open source project ecosystems. According to early data from Sonatype s 8th annual State of the Software Supply Chain Report, which will be released in full this October, Sonatype has recorded an average 700% jump in repository attacks over the last three years.

More information is available in the press release.

A number of changes were made to the Reproducible Builds website and documentation this month, including Chris Lamb adding a redirect from /projects/ to /who/ in order to keep old or archived links working [ ], Jelle van der Waa added a Rust programming language example for SOURCE_DATE_EPOCH [ ][ ] and Mattia Rizzolo included Protocol Labs amongst our project-level sponsors [ ].

Debian There was a large amount of reproducibility work taking place within Debian this month:

The `nfft` source package was removed from the archive, and now all packages in Debian bookworm now have a corresponding `.buildinfo` file. This can be confirmed and tracked on the associated page on the tests.reproducible-builds.org site.

Vagrant Cascadian announced on our mailing list an informal online sprint to help clear the huge backlog of reproducible builds patches submitted by performing NMU (Non-Maintainer Uploads). The first such sprint took place on September 22nd with the following results:

Holger Levsen:

Mailed #1010957 in `man-db` asking for an update and whether to remove the patch tag for now. This was subsequently removed and the maintainer started to address the issue.

Uploaded `gmp` to `DELAYED/15`, fixing #1009931.

Emailed #1017372 in `plymouth` and asked for the maintainer s opinion on the patch. This resulted in the maintainer improving Vagrant s original patch (and uploading it) as well as filing an issue upstream.

Uploaded `time` to `DELAYED/15`, fixing #983202.

Vagrant Cascadian:

Verify and updated patch for `mylvmbackup` (#782318)

Verified/updated patches for `libranlip`. (#788000, #846975 & #1007137)

Uploaded `libranlip` to `DELAYED/10`.

Verified patch for `cclive`. (#824501)

Uploaded `cclive` to `DELAYED/10`.

Vagrant was unable to reproduce the underlying issue within #791423 (`linuxtv-dvb-apps`) and so the bug was marked as done .

Researched #794398 (in `clhep`).

The plan is to repeat these sprints every two weeks, with the next taking place on Thursday October 6th at 16:00 UTC on the `#debian-reproducible` IRC channel.

Roland Clobus posted his 13th update of the status of reproducible Debian ISO images on our mailing list. During the last month, Roland ensured that the live images are now automatically fed to openQA for automated testing after they have been shown to be reproducible. Additionally Roland asked on the debian-devel mailing list about a way to determine the canonical timestamp of the Debian archive. [ ]

Following up on last month s work on reproducible bootstrapping, Holger Levsen filed two bugs against the debootstrap and cdebootstrap utilities. (#1019697 & #1019698)

Lastly, 44 reviews of Debian packages were added, 91 were updated and 17 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated too, including the descriptions of `cmake_rpath_contains_build_path` [ ], `nondeterministic_version_generated_by_python_param` [ ] and `timestamps_in_documentation_generated_by_org_mode` [ ]. Furthermore, two new issue types were created: `build_path_used_to_determine_version_or_package_name` [ ] and `captures_build_path_via_cmake_variables` [ ].

Other distributions In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `222` and `223` to Debian, as well as made the following changes:

The `cbfstools` utility is now provided in Debian via the `coreboot-utils` package so we can enable that functionality within Debian. [ ]

Looked into Mach-O support.

Fixed the try.diffoscope.org service by addressing a compatibility issue between `glibc`/`seccomp` that was preventing the Docker-contained diffoscope instance from spawning any external processes whatsoever [ ]. I also updated the `requirements.txt` file, as some of the specified packages were no longer available [ ][ ].

In addition Jelle van der Waa added support for `file` version 5.43 [ ] and Mattia Rizzolo updated the packaging:

Also include `coreboot-utils` in the `Build-Depends` and `Test-Depends` fields so that it is available for tests. [ ]

Use pep517 and pip to load the requirements. [ ]

Remove packages in `Breaks`/`Replaces` that have been obsoleted since the release of Debian bullseye. [ ]

Reprotest reprotest is our end-user tool to build the same source code twice in widely and deliberate different environments, and checking whether the binaries produced by the builds have any differences. This month, reprotest version `0.7.22` was uploaded to Debian unstable by Holger Levsen, which included the following changes by Philip Hands:

Actually ensure that the `setarch(8)` utility can actually execute before including an architecture to test. [ ]

Include all files matching `.deb` in the default `artifact_pattern` in order to archive all results of the build. [ ]

Emit an error when building the Debian package if the Debian packaging version does not patch the Python version of reprotest. [ ]

Remove an unneeded invocation of the `head(1)` utility. [ ]

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann (18 bugs):

`DateTime` (fails to build in 2038)

`FreeRCT` (date-related issue)

`clanlib1` (filesystem ordering)

`cli` (fails to build in 2038)

`deepin-gettext-tools` (patch+version update toolchain sort python glob)

`mariadb` (fails to build in 2038)

`mercurial` (fails to build in 2038)

`mirrormagic` (parallelism-related issue)

`ocaml-extlib` (parallelism-related issue)

`python-xmlrpc/python-softlayer` (fails to build in 2038)

`python` (fails to build in 2038)

`q3rally` (zip-related issue)

`rnd_jue` (parallelism-related issue)

`rsync` (workaround an issue in GCC 7.x)

`scons` (`SOURCE_DATE_EPOCH`-related issue)

`stratagus` (date-related issue)

`triplane` (nondeterminism caused by uninitialised memory)

`tyrquake` (date-related issue)

Chris Lamb:

#1019382 filed against `gnome-online-accounts`.

There was renewed activity on a reproducibility-related bug in the Sphinx documentation tool this month. Originally filed in October 2021 by Chris Lamb, the bug in question relates to contents of the `LANGUAGE` environment variable inconsistently affecting the output of `objects.inv` files.

Jelle van der Waa:

`mp4v2` (date-related issue)

`mm-common` (uid/gid issue)

`aardvark-dns` (date-related issue)

Vagrant Cascadian (70 bugs!):

#1020648 filed against `extrepo-data`.

#1020650 filed against `tmpreaper`.

#1020651 filed against `xmlrpc-epi`.

#1020653 filed against `pal`.

#1020656 filed against `nvram-wakeup`.

#1020657 filed against `netris`.

#1020658 filed against `netpbm-free`.

#1020659 filed against `lookup`.

#1020660 filed against `logtools`.

#1020661 filed against `libid3tag`.

#1020662 filed against `log4cpp`.

#1020665 filed against `libimage-imlib2-perl`.

#1020668 filed against `jnettop`.

#1020670 filed against `gwaei`.

#1020671 filed against `ipfm`.

#1020672 filed against `tarlz`.

#1020673 filed against `w3cam`.

#1020674 filed against `ifstat`.

#1020715 filed against `xserver-xorg-input-joystick`.

#1020719 filed against `chibicc`.

#1020723 filed against `python-omegaconf`.

#1020724 and #1020725 filed against `snapper`.

#1020736 filed against `libreswan`.

#1020743 filed against `pure-ftpd`.

#1020748 filed against `xcolmix`.

#1020749 filed against `gigalomania`.

#1020750 filed against `xjump`.

#1020751 filed against `waili`.

#1020752 filed against `sjeng`.

#1020753 filed against `seqtk`.

#1020754 filed against `shapetools`.

#1020755 filed against `rotter`.

#1020756 filed against `rakarrack`.

#1020757 filed against `rig`.

#1020759 filed against `postal`.

#1020798 filed against `netkit-rsh`.

#1020800 filed against `libapache-mod-evasive`.

#1020804 filed against `paxctl`.

#1020805 filed against `png23d`.

#1020806 filed against `perl-byacc`.

#1020807 filed against `poster`.

#1020808 filed against `powerdebug`.

#1020809 filed against `aespipe`.

#1020810 filed against `aewm++-goodies`.

#1020811 filed against `apache-upload-progress-module`.

#1020812 filed against `ascii2binary`.

#1020813 filed against `bible-kjv`.

#1020814 filed against `dradio`.

#1020815 filed against `libapache2-mod-python`.

#1020816 filed against `tempest-for-eliza`.

#1020817 filed against `aplus-fsf`.

#1020866 filed against `wrapsrv`.

#1020867 filed against `uclibc`.

#1020870 filed against `xppaut`.

#1020872 filed against `xvier`.

#1020873 filed against `xserver-xorg-video-glide`.

#1020875 filed against `z80asm`.

#1020876 filed against `yaskkserv`.

#1020877 filed against `edid-decode`.

#1020878 filed against `dustmite`.

#1020879 filed against `dustmite`.

#1020880 filed against `libapache2-mod-authnz-pam`.

#1020881 filed against `kafs-client`.

#1020882 filed against `yaku-ns`.

#1020884 filed against `bplay`.

#1020886 filed against `chise-base`.

#1020887 filed against `checkpw`.

#1020888 filed against `clamz`.

#1020889 filed against `libapache2-mod-auth-pgsql`.

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. This month, however, the following changes were made:

Holger Levsen:

Add a job to build reprotest from Git [ ] and use the correct Git branch when building it [ ].

Mattia Rizzolo:

Enable syncing of results from building live Debian ISO images. [ ]

Use `scp -p` in order to preserve modification times when syncing live ISO images. [ ]

Apply the shellcheck shell script analysis tool. [ ]

In a build node wrapper script, remove some debugging code which was messing up calling `scp(1)` correctly [ ] and consquently add support to use both `scp -p` and regular `scp` [ ].

Roland Clobus:

Track and handle the case where the Debian archive gets updated between two live image builds. [ ]

Remove a call to `sudo(1)` as it is not (or no longer) required to delete old live-build results. [ ]

Contact As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

29 September 2022

Antoine Beaupr : Detecting manual (and optimizing large) package installs in Puppet

Well this is a mouthful. I recently worked on a neat hack called puppet-package-check. It is designed to warn about manually installed packages, to make sure "everything is in Puppet". But it turns out it can (probably?) dramatically decrease the bootstrap time of Puppet bootstrap when it needs to install a large number of packages.

Detecting manual packages On a cleanly filed workstation, it looks like this:

root@emma:/home/anarcat/bin# ./puppet-package-check -v
listing puppet packages...
listing apt packages...
loading apt cache...
0 unmanaged packages found

A messy workstation will look like this:

root@curie:/home/anarcat/bin# ./puppet-package-check -v
listing puppet packages...
listing apt packages...
loading apt cache...
288 unmanaged packages found
apparmor-utils beignet-opencl-icd bridge-utils clustershell cups-pk-helper davfs2 dconf-cli dconf-editor dconf-gsettings-backend ddccontrol ddrescueview debmake debootstrap decopy dict-devil dict-freedict-eng-fra dict-freedict-eng-spa dict-freedict-fra-eng dict-freedict-spa-eng diffoscope dnsdiag dropbear-initramfs ebtables efibootmgr elpa-lua-mode entr eog evince figlet file file-roller fio flac flex font-manager fonts-cantarell fonts-inconsolata fonts-ipafont-gothic fonts-ipafont-mincho fonts-liberation fonts-monoid fonts-monoid-tight fonts-noto fonts-powerline fonts-symbola freeipmi freetype2-demos ftp fwupd-amd64-signed gallery-dl gcc-arm-linux-gnueabihf gcolor3 gcp gdisk gdm3 gdu gedit gedit-plugins gettext-base git-debrebase gnome-boxes gnote gnupg2 golang-any golang-docker-credential-helpers golang-golang-x-tools grub-efi-amd64-signed gsettings-desktop-schemas gsfonts gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-pulseaudio gtypist gvfs-backends hackrf hashcat html2text httpie httping hugo humanfriendly iamerican-huge ibus ibus-gtk3 ibus-libpinyin ibus-pinyin im-config imediff img2pdf imv initramfs-tools input-utils installation-birthday internetarchive ipmitool iptables iptraf-ng jackd2 jupyter jupyter-nbextension-jupyter-js-widgets jupyter-qtconsole k3b kbtin kdialog keditbookmarks keepassxc kexec-tools keyboard-configuration kfind konsole krb5-locales kwin-x11 leiningen lightdm lintian linux-image-amd64 linux-perf lmodern lsb-base lvm2 lynx lz4json magic-wormhole mailscripts mailutils manuskript mat2 mate-notification-daemon mate-themes mime-support mktorrent mp3splt mpdris2 msitools mtp-tools mtree-netbsd mupdf nautilus nautilus-sendto ncal nd ndisc6 neomutt net-tools nethogs nghttp2-client nocache npm2deb ntfs-3g ntpdate nvme-cli nwipe obs-studio okular-extra-backends openstack-clients openstack-pkg-tools paprefs pass-extension-audit pcmanfm pdf-presenter-console pdf2svg percol pipenv playerctl plymouth plymouth-themes popularity-contest progress prometheus-node-exporter psensor pubpaste pulseaudio python3-ldap qjackctl qpdfview qrencode r-cran-ggplot2 r-cran-reshape2 rake restic rhash rpl rpm2cpio rs ruby ruby-dev ruby-feedparser ruby-magic ruby-mocha ruby-ronn rygel-playbin rygel-tracker s-tui sanoid saytime scrcpy scrcpy-server screenfetch scrot sdate sddm seahorse shim-signed sigil smartmontools smem smplayer sng sound-juicer sound-theme-freedesktop spectre-meltdown-checker sq ssh-audit sshuttle stress-ng strongswan strongswan-swanctl syncthing system-config-printer system-config-printer-common system-config-printer-udev systemd-bootchart systemd-container tardiff task-desktop task-english task-ssh-server tasksel tellico texinfo texlive-fonts-extra texlive-lang-cyrillic texlive-lang-french texlive-lang-german texlive-lang-italian texlive-xetex tftp-hpa thunar-archive-plugin tidy tikzit tint2 tintin++ tipa tpm2-tools traceroute tree trocla ucf udisks2 unifont unrar-free upower usbguard uuid-runtime vagrant-cachier vagrant-libvirt virt-manager vmtouch vorbis-tools w3m wamerican wamerican-huge wfrench whipper whohas wireshark xapian-tools xclip xdg-user-dirs-gtk xlax xmlto xsensors xserver-xorg xsltproc xxd xz-utils yubioath-desktop zathura zathura-pdf-poppler zenity zfs-dkms zfs-initramfs zfsutils-linux zip zlib1g zlib1g-dev
157 old: apparmor-utils clustershell davfs2 dconf-cli dconf-editor ddccontrol ddrescueview decopy dnsdiag ebtables efibootmgr elpa-lua-mode entr figlet file-roller fio flac flex font-manager freetype2-demos ftp gallery-dl gcc-arm-linux-gnueabihf gcolor3 gcp gdu gedit git-debrebase gnote golang-docker-credential-helpers golang-golang-x-tools gtypist hackrf hashcat html2text httpie httping hugo humanfriendly iamerican-huge ibus ibus-pinyin imediff input-utils internetarchive ipmitool iptraf-ng jackd2 jupyter-qtconsole k3b kbtin kdialog keditbookmarks keepassxc kexec-tools kfind konsole leiningen lightdm lynx lz4json magic-wormhole manuskript mat2 mate-notification-daemon mktorrent mp3splt msitools mtp-tools mtree-netbsd nautilus nautilus-sendto nd ndisc6 neomutt net-tools nethogs nghttp2-client nocache ntpdate nwipe obs-studio openstack-pkg-tools paprefs pass-extension-audit pcmanfm pdf-presenter-console pdf2svg percol pipenv playerctl qjackctl qpdfview qrencode r-cran-ggplot2 r-cran-reshape2 rake restic rhash rpl rpm2cpio rs ruby-feedparser ruby-magic ruby-mocha ruby-ronn s-tui saytime scrcpy screenfetch scrot sdate seahorse shim-signed sigil smem smplayer sng sound-juicer spectre-meltdown-checker sq ssh-audit sshuttle stress-ng system-config-printer system-config-printer-common tardiff tasksel tellico texlive-lang-cyrillic texlive-lang-french tftp-hpa tikzit tint2 tintin++ tpm2-tools traceroute tree unrar-free vagrant-cachier vagrant-libvirt vmtouch vorbis-tools w3m wamerican wamerican-huge wfrench whipper whohas xdg-user-dirs-gtk xlax xmlto xsensors xxd yubioath-desktop zenity zip
131 new: beignet-opencl-icd bridge-utils cups-pk-helper dconf-gsettings-backend debmake debootstrap dict-devil dict-freedict-eng-fra dict-freedict-eng-spa dict-freedict-fra-eng dict-freedict-spa-eng diffoscope dropbear-initramfs eog evince file fonts-cantarell fonts-inconsolata fonts-ipafont-gothic fonts-ipafont-mincho fonts-liberation fonts-monoid fonts-monoid-tight fonts-noto fonts-powerline fonts-symbola freeipmi fwupd-amd64-signed gdisk gdm3 gedit-plugins gettext-base gnome-boxes gnupg2 golang-any grub-efi-amd64-signed gsettings-desktop-schemas gsfonts gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-pulseaudio gvfs-backends ibus-gtk3 ibus-libpinyin im-config img2pdf imv initramfs-tools installation-birthday iptables jupyter jupyter-nbextension-jupyter-js-widgets keyboard-configuration krb5-locales kwin-x11 lintian linux-image-amd64 linux-perf lmodern lsb-base lvm2 mailscripts mailutils mate-themes mime-support mpdris2 mupdf ncal npm2deb ntfs-3g nvme-cli okular-extra-backends openstack-clients plymouth plymouth-themes popularity-contest progress prometheus-node-exporter psensor pubpaste pulseaudio python3-ldap ruby ruby-dev rygel-playbin rygel-tracker sanoid scrcpy-server sddm smartmontools sound-theme-freedesktop strongswan strongswan-swanctl syncthing system-config-printer-udev systemd-bootchart systemd-container task-desktop task-english task-ssh-server texinfo texlive-fonts-extra texlive-lang-german texlive-lang-italian texlive-xetex thunar-archive-plugin tidy tipa trocla ucf udisks2 unifont upower usbguard uuid-runtime virt-manager wireshark xapian-tools xclip xserver-xorg xsltproc xz-utils zathura zathura-pdf-poppler zfs-dkms zfs-initramfs zfsutils-linux zlib1g zlib1g-dev

Yuck! That's a lot of shit to go through. Notice how the packages get sorted between "old" and "new" packages. This is because popcon is used as a tool to mark which packages are "old". If you have unmanaged packages, the "old" ones are likely things that you can uninstall, for example. If you don't have popcon installed, you'll also get this warning:

popcon stats not available: [Errno 2] No such file or directory: '/var/log/popularity-contest'

The error can otherwise be safely ignored, but you won't get "help" prioritizing the packages to add to your manifests. Note that the tool ignores packages that were "marked" (see apt-mark(8)) as automatically installed. This implies that you might have to do a little bit of cleanup the first time you run this, as Debian doesn't necessarily mark all of those packages correctly on first install. For example, here's how it looks like on a clean install, after Puppet ran:

root@angela:/home/anarcat# ./bin/puppet-package-check -v
listing puppet packages...
listing apt packages...
loading apt cache...
127 unmanaged packages found
ca-certificates console-setup cryptsetup-initramfs dbus file gcc-12-base gettext-base grub-common grub-efi-amd64 i3lock initramfs-tools iw keyboard-configuration krb5-locales laptop-detect libacl1 libapparmor1 libapt-pkg6.0 libargon2-1 libattr1 libaudit-common libaudit1 libblkid1 libbpf0 libbsd0 libbz2-1.0 libc6 libcap-ng0 libcap2 libcap2-bin libcom-err2 libcrypt1 libcryptsetup12 libdb5.3 libdebconfclient0 libdevmapper1.02.1 libedit2 libelf1 libext2fs2 libfdisk1 libffi8 libgcc-s1 libgcrypt20 libgmp10 libgnutls30 libgpg-error0 libgssapi-krb5-2 libhogweed6 libidn2-0 libip4tc2 libiw30 libjansson4 libjson-c5 libk5crypto3 libkeyutils1 libkmod2 libkrb5-3 libkrb5support0 liblocale-gettext-perl liblockfile-bin liblz4-1 liblzma5 libmd0 libmnl0 libmount1 libncurses6 libncursesw6 libnettle8 libnewt0.52 libnftables1 libnftnl11 libnl-3-200 libnl-genl-3-200 libnl-route-3-200 libnss-systemd libp11-kit0 libpam-systemd libpam0g libpcre2-8-0 libpcre3 libpcsclite1 libpopt0 libprocps8 libreadline8 libselinux1 libsemanage-common libsemanage2 libsepol2 libslang2 libsmartcols1 libss2 libssl1.1 libssl3 libstdc++6 libsystemd-shared libsystemd0 libtasn1-6 libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl libtinfo6 libtirpc-common libtirpc3 libudev1 libunistring2 libuuid1 libxtables12 libxxhash0 libzstd1 linux-image-amd64 logsave lsb-base lvm2 media-types mlocate ncurses-term pass-extension-otp puppet python3-reportbug shim-signed tasksel ucf usr-is-merged util-linux-extra wpasupplicant xorg zlib1g
popcon stats not available: [Errno 2] No such file or directory: '/var/log/popularity-contest'

Normally, there should be unmanaged packages here. But because of the way Debian is installed, a lot of libraries and some core packages are marked as manually installed, and are of course not managed through Puppet. There are two solutions to this problem:

really manage everything in Puppet (argh)
mark packages as automatically installed

I typically chose the second path and mark a ton of stuff as automatic. Then either they will be auto-removed, or will stop being listed. In the above scenario, one could mark all libraries as automatically installed with:

apt-mark auto $(./bin/puppet-package-check   grep -o 'lib[^ ]*')

... but if you trust that most of that stuff is actually garbage that you don't really want installed anyways, you could just mark it all as automatically installed:

apt-mark auto $(./bin/puppet-package-check)

In my case, that ended up keeping basically all libraries (because of course they're installed for some reason) and auto-removing this:

dh-dkms discover-data dkms libdiscover2 libjsoncpp25 libssl1.1 linux-headers-amd64 mlocate pass-extension-otp pass-otp plocate x11-apps x11-session-utils xinit xorg

You'll notice xorg in there: yep, that's bad. Not what I wanted. But for some reason, on other workstations, I did not actually have xorg installed. Turns out having xserver-xorg is enough, and that one has dependencies. So now I guess I just learned to stop worrying and live without X(org).

Optimizing large package installs But that, of course, is not all. Why make things simple when you can have an unreadable title that is trying to be both syntactically correct and click-baity enough to flatter my vain ego? Right. One of the challenges in bootstrapping Puppet with large package lists is that it's slow. Puppet lists packages as individual resources and will basically run apt install $PKG on every package in the manifest, one at a time. While the overhead of apt is generally small, when you add things like apt-listbugs, apt-listchanges, needrestart, triggers and so on, it can take forever setting up a new host. So for initial installs, it can actually makes sense to skip the queue and just install everything in one big batch. And because the above tool inspects the packages installed by Puppet, you can run it against a catalog and have a full lists of all the packages Puppet would install, even before I even had Puppet running. So when reinstalling my laptop, I basically did this:

apt install puppet-agent/experimental
puppet agent --test --noop
apt install $(./puppet-package-check --debug \
    2>&1   grep ^puppet\ packages 
      sed 's/puppet packages://;s/ /\n/g'
      grep -v -e onionshare -e golint -e git-sizer -e github-backup -e hledger -e xsane -e audacity -e chirp -e elpa-flycheck -e elpa-lsp-ui -e yubikey-manager -e git-annex -e hopenpgp-tools -e puppet
) puppet-agent/experimental

That massive grep was because there are currently a lot of packages missing from bookworm. Those are all packages that I have in my catalog but that still haven't made it to bookworm. Sad, I know. I eventually worked around that by adding bullseye sources so that the Puppet manifest actually ran. The point here is that this improves the Puppet run time a lot. All packages get installed at once, and you get a nice progress bar. Then you actually run Puppet to deploy configurations and all the other goodies:

puppet agent --test

I wish I could tell you how much faster that ran. I don't know, and I will not go through a full reinstall just to please your curiosity. The only hard number I have is that it installed 444 packages (which exploded in 10,191 packages with dependencies) in a mere 10 minutes. That might also be with the packages already downloaded. In any case, I have that gut feeling it's faster, so you'll have to just trust my gut. It is, after all, much more important than you might think.

Similar work The blueprint system is something similar to this:

It figures out what you ve done manually, stores it locally in a Git repository, generates code that s able to recreate your efforts, and helps you deploy those changes to production

That tool has unfortunately been abandoned for a decade at this point. Also note that the AutoRemove::RecommendsImportant and AutoRemove::SuggestsImportant are relevant here. If it is set to true (the default), a package will not be removed if it is (respectively) a Recommends or Suggests of another package (as opposed to the normal Depends). In other words, if you want to also auto-remove packages that are only Suggests, you would, for example, add this to apt.conf:

AutoRemove::SuggestsImportant false;

Paul Wise has tried to make the Debian installer and debootstrap properly mark packages as automatically installed in the past, but his bug reports were rejected. The other suggestions in this section are also from Paul, thanks!

9 September 2022

Reproducible Builds: Reproducible Builds in August 2022

Welcome to the August 2022 report from the Reproducible Builds project! In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Community news As announced last month, registration is currently open for our in-person summit this year which is due to be held between November 1st November 3rd. The event will take place in Venice (Italy). Very soon we intend to pick a venue reachable via the train station and an international airport. However, the precise venue will depend on the number of attendees. Please see the announcement email for information about how to register.
The US National Security Agency (NSA), Cybersecurity and Infrastructure Security Agency (CISA) and the Office of the Director of National Intelligence (ODNI) have released a document called Securing the Software Supply Chain: Recommended Practices Guide for Developers (PDF) as part of their Enduring Security Framework (ESF) work. The document expressly recommends having reproducible builds as part of advanced recommended mitigations, along with hermetic builds. Page 31 (page 35 in the PDF) says:
Reproducible builds provide additional protection and validation against attempts to compromise build systems. They ensure the binary products of each build system match: i.e., they are built from the same source, regardless of variable metadata such as the order of input files, timestamps, locales, and paths. Reproducible builds are those where re-running the build steps with identical input artifacts results in bit-for-bit identical output. Builds that cannot meet this must provide a justification why the build cannot be made reproducible.
The full press release is available online.
On our mailing list this month, Marc Prud hommeaux posted a feature request for diffoscope which additionally outlines a project called The App Fair, an autonomous distribution network of free and open-source macOS and iOS applications, where validated apps are then signed and submitted for publication .
Author/blogger Cory Doctorow posted published a provocative blog post this month titled Your computer is tormented by a wicked god . Touching on Ken Thompson s famous talk, Reflections on Trusting Trust , the early goals of Secure Computing and UEFI firmware interfaces:
This is the core of a two-decade-old debate among security people, and it s one that the benevolent God faction has consistently had the upper hand in. They re the curated computing advocates who insist that preventing you from choosing an alternative app store or side-loading a program is for your own good because if it s possible for you to override the manufacturer s wishes, then malicious software may impersonate you to do so, or you might be tricked into doing so. [..] This benevolent dictatorship model only works so long as the dictator is both perfectly benevolent and perfectly competent. We know the dictators aren t always benevolent. [ ] But even if you trust a dictator s benevolence, you can t trust in their perfection. Everyone makes mistakes. Benevolent dictator computing works well, but fails badly. Designing a computer that intentionally can t be fully controlled by its owner is a nightmare, because that is a computer that, once compromised, can attack its owner with impunity.

Lastly, Chengyu HAN updated the Reproducible Builds website to correct an incorrect Git command. [ ]

Debian

In Debian this month, the essential and required package sets became 100% reproducible in Debian bookworm on the amd64 and arm64 architectures. These two subsets of the full Debian archive refer to Debian package priority levels as described in the 2.5 Priorities section of the Debian Policy there is no canonical minimal installation package set in Debian due to its diverse methods of installation. As it happens, these package sets are not reproducible on the i386 architecture because the ncurses package on that architecture is not yet reproducible, and the sed package currently fails to build from source on armhf too. The full list of reproducible packages within these package sets can be viewed within our QA system, such as on the page of required packages in amd64 and the list of essential packages on arm64, both for Debian bullseye.

It recently has become very easy to install reproducible Debian Docker containers using podman on Debian bullseye:

$ sudo apt install podman
$ podman run --rm -it debian:bullseye bash

The (pre-built) image used is itself built using debuerrotype, as explained on docker.debian.net. This page also details how to build the image yourself and what checksums are expected if you do so.

Related to this, it has also become straightforward to reproducibly bootstrap Debian using mmdebstrap, a replacement for the usual debootstrap tool to create Debian root filesystems:

$ SOURCE_DATE_EPOCH=$(date --utc --date=2022-08-29 +%s) mmdebstrap unstable > unstable.tar

This works for (at least) Debian unstable, bullseye and bookworm, and is tested automatically by a number of QA jobs set up by Holger Levsen (unstable, bookworm and bullseye)

Work has also taken place to ensure that the canonical debootstrap and cdebootstrap tools are also capable of bootstrapping Debian reproducibly, although it currently requires a few extra steps:

Clamping the modification time of files that are newer than $SOURCE_DATE_EPOCH to be not greater than SOURCE_DATE_EPOCH.
Deleting a few files. For debootstrap, this requires the deletion of /etc/machine-id, /var/cache/ldconfig/aux-cache, /var/log/dpkg.log, /var/log/alternatives.log and /var/log/bootstrap.log, and for cdebootstrap we also need to delete the /var/log/apt/history.log and /var/log/apt/term.log files as well.

This process works at least for unstable, bullseye and bookworm and is now being tested automatically by a number of QA jobs setup by Holger Levsen [ ][ ][ ][ ][ ][ ]. As part of this work, Holger filed two bugs to request a better initialisation of the /etc/machine-id file in both debootstrap [ ] and cdebootstrap [ ].

Elsewhere in Debian, 131 reviews of Debian packages were added, 20 were updated and 27 were removed this month, adding to our extensive knowledge about identified issues. Chris Lamb added a number of issue types, including: randomness_in_browserify_output [ ], haskell_abi_hash_differences [ ], nondeterministic_ids_in_html_output_generated_by_python_sphinx_panels [ ]. Lastly, Mattia Rizzolo removed the deterministic flag from the captures_kernel_variant flag [ ].

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `220` and `221` to Debian, as well as made the following changes:

Update `external_tools.py` to reflect changes to `xxd` and the `vim-common` package. [ ]

Depend on the dedicated `xxd` package now, not the `vim-common` package. [ ]

Don t crash if we can open a PDF file using the PyPDF library, but cannot subsequently parse the annotations within. [ ]

In addition, Vagrant Cascadian updated diffoscope in GNU Guix, first to to version 220 [ ] and later to 221 [ ].

Community news The Reproducible Builds project aims to fix as many currently-unreproducible packages as possible as well as to send all of our patches upstream wherever appropriate. This month we created a number of patches, including:

Bernhard M. Wiedemann:

`at-spi-sharp` (build failure when build on a multiprocessor machine).

`borgbackup` (fails to build in 2038, fix)

`buzztrax` (parallelism-related issue)

`chart-testing` (date-related issue)

`memcached` (fails to build in 2038)

`nim` (fails to build in 2038)

`perl-Time-Moment` (fails to build in 2038)

`python-bson` (fails to build in 2038)

`python-heatclient` (fails to build in 2038)

`python3.8` (fails to build in 2038)

`reproducible-faketools`

`s3fs` (date-related issue)

`systemd` (date-related issue)

Chris Lamb:

#1016486 filed against `wayfire`.

#1016583 filed against `multipath-tools`.

#1016584 filed against `node-canvas-confetti`.

#1017473 filed against `psi` (forwarded upstream).

#1017475 filed against `sphinx-panels` (forwarded upstream).

#1017920 filed against `sysfsutils`.

#1017945 filed against `geeqie`.

Vagrant Cascadian:

#1017073 filed against `python-suntime`.

#1017372 filed against `plymouth`.

#1017373 filed against `tiemu`.

#1017421 filed against `fltk1.3`.

#1018802 filed against `localechooser`.

`uboot` (Rasmus Villemoes proposed fixing gcc instead [ ][ ])

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, Holger Levsen made the following changes:

Debian-related changes:

Temporarily add Debian unstable `deb-src` lines to enable test builds a Non-maintainer Upload (NMU) campaign targeting 708 sources without `.buildinfo` files found in Debian unstable, including 475 in bookworm. [ ][ ]

Correctly deal with the Debian Edu packages not being installable. [ ]

Finally, stop scheduling stretch. [ ]

Make sure all Ubuntu nodes have the `linux-image-generic` kernel package installed. [ ]

Health checks & view:

Detect SSH login problems. [ ]

Only report the first uninstallable package set. [ ]

Show new bootstrap jobs. [ ] and debian-live jobs. [ ] in the job health view.

Fix regular expression to detect various zombie jobs. [ ]

New jobs:

Add a new job to test reproducibility of mmdebstrap bootstrapping tool. [ ][ ][ ][ ]

Run our new mmdebstrap job remotely [ ][ ]

Improve the output of the mmdebstrap job. [ ][ ][ ]

Adjust the mmdebstrap script to additionally support debootstrap as well. [ ][ ][ ]

Work around mmdebstrap and debootstrap keeping logfiles within their artifacts. [ ][ ][ ]

Add support for testing cdebootstrap too and add such a job for unstable. [ ][ ][ ]

Use a reproducible value for `SOURCE_DATE_EPOCH` for all our new bootstrap jobs. [ ]

Misc changes:

Send the `create_meta_pkg_sets` notification to `#debian-reproducible-changes` instead of `#debian-reproducible`. [ ]

In addition, Roland Clobus re-enabled the tests for live-build images [ ] and added a feature where the build would retry instead of give up when the archive was synced whilst building an ISO [ ], and Vagrant Cascadian added logging to report the current target of the `/bin/sh` symlink [ ].

Contact As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

4 August 2022

Reproducible Builds: Reproducible Builds in July 2022

Welcome to the July 2022 report from the Reproducible Builds project! In our reports we attempt to outline the most relevant things that have been going on in the past month. As a brief introduction, the reproducible builds effort is concerned with ensuring no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Reproducible Builds summit 2022 Despite several delays, we are pleased to announce that registration is open for our in-person summit this year: November 1st November 3rd
The event will happen in Venice (Italy). We intend to pick a venue reachable via the train station and an international airport. However, the precise venue will depend on the number of attendees. Please see the announcement email for information about how to register.

Is reproducibility practical? Ludovic Court s published an informative blog post this month asking the important question: Is reproducibility practical?:
Our attention was recently caught by a nice slide deck on the methods and tools for reproducible research in the R programming language. Among those, the talk mentions Guix, stating that it is for professional, sensitive applications that require ultimate reproducibility , which is probably a bit overkill for Reproducible Research . While we were flattered to see Guix suggested as good tool for reproducibility, the very notion that there s a kind of reproducibility that is ultimate and, essentially, impractical, is something that left us wondering: What kind of reproducibility do scientists need, if not the ultimate kind? Is reproducibility practical at all, or is it more of a horizon?
The post goes on to outlines the concept of reproducibility, situating examples within the context of the GNU Guix operating system.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `218`, `219` and `220` to Debian, as well as made the following changes:

New features:

Support Haskell 9.x series files. [ ]

Bug fixes:

Fix a regression introduced in version 207 where diffoscope would crash if one directory contained a directory that wasn t in the other. Thanks to Alderico Gallo for the testcase. [ ]

Don t traceback if we encounter an invalid Unicode character in Haskell versioning headers. [ ]

Output improvements:

Improve output of Markdown and reStructuredText to use code blocks with highlighting. [ ]

Codebase improvements:

Space out a file a little. [ ]

Update various copyright years. [ ]

Mailing list On our mailing list this month:

Roland Clobus posted his Eleventh status update about reproducible [Debian] live-build ISO images, noting amongst many other things! that all major desktops build reproducibly with bullseye, bookworm and sid.

Santiago Torres-Arias announced a Call for Papers (CfP) for a new SCORED conference, an academic workshop around software supply chain security . As Santiago highlights, this new conference invites reviewers from industry, open source, governement and academia to review the papers [and] I think that this is super important to tackle the supply chain security task .

Upstream patches The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. This month, however, we submitted the following patches:

Bernhard M. Wiedemann

openSUSE monthly report

`acarsdec` (embeds CPU info with `march=native`)

`casacore` (embeds CPU info with `march=native`)

`kubernetes` (uses random name of temporary directory)

`setuptools/python-brotlicffi` (toolchain, filesys/readdir)

`sysstat` (FTBFS in single CPU mode)

`sundials` (FTBFS in single CPU mode)

`nim` (FTBFS in single CPU mode)

`doxygen/libzypp` (toolchain readdir)

`python-pyquil` (build failure)

`openssl-1_0_0` (build failure)

`jsonrpc-glib` (FTBFS in single CPU mode)

`slurm` (Link-Time Optimisation and `.tar` issues)

`wasi-libc` (sort the output from `find`)

Chris Lamb:

#1015245 filed against `libshumate`.

#1015246 filed against `zeal`.

#1016186 filed against `gappa`.

Philip Rinn:

#1014877 filed against `cxref`.

Vagrant Cascadian:

#1014426 filed against `cldump`.

#1014428 filed against `xmacro`.

#1014559 filed against `libloki`.

#1014560 filed against `ygl`.

#1014561 filed against `clp`.

#1014564 filed against `tvtime`.

#1014789 filed against `rdiff-backup-fs`.

Reprotest reprotest is the Reproducible Builds project s end-user tool to build the same source code twice in widely and deliberate different environments, and checking whether the binaries produced by the builds have any differences. This month, the following changes were made:

Holger Levsen:

Uploaded version `0.7.21` to Debian unstable as well as mark `0.7.22` development in the repository [ ].

Make diffoscope dependency unversioned as the required version is met even in Debian buster. [ ]

Revert an accidentally committed hunk. [ ]

Mattia Rizzolo:

Apply a patch from Nick Rosbrook to not force the tests to run only against Python 3.9. [ ]

Run the tests through `pybuild` in order to run them against all supported Python 3.x versions. [ ]

Fix a deprecation warning in the `setup.cfg` file. [ ]

Close a new Debian bug. [ ]

Reproducible builds website A number of changes were made to the Reproducible Builds website and documentation this month, including:

Arnout Engelen:

Add a link to recent May Contain Hackers 2022 conference talk slides. [ ]

Chris Lamb:

Correct some grammar. [ ]

Holger Levsen:

Add talk from FOSDEM 2015 presented by Holger and Lunar. [ ]

Show date of presentations if we have them. [ ][ ]

Add my presentation from DebConf22 [ ] and from Debian Reunion Hamburg 2022 [ ].

Add dhole to the speakers of the DebConf15 talk. [ ]

Add raboof s talk Reproducible Builds for Trustworthy Binaries from May Contain Hackers. [ ]

Drop some Debian-related suggested ideas which are not really relevant anymore. [ ]

Add a link to list of packages with patches ready to be NMUed. [ ]

Mattia Rizzolo:

Add information about our upcoming event in Venice. [ ][ ][ ][ ]

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, Holger Levsen made the following changes:

Debian-related changes:

Create graphs displaying existing `.buildinfo` files per each Debian suite/arch. [ ][ ]

Fix a typo in the Debian dashboard. [ ][ ]

Fix some issues in the `pkg-r` package set definition. [ ][ ][ ]

Improve the builtin-pho HTML output. [ ][ ][ ][ ]

Temporarily disable all live builds as our snapshot mirror is offline. [ ]

Automated node health checks:

Detect `dpkg` failures. [ ]

Detect files with bad UNIX permissions. [ ]

Relax a regular expression in order to detect Debian Live image build failures. [ ]

Misc changes:

Test that FreeBSD virtual machine has been updated to version 13.1. [ ]

Add a reminder about powercycling the `armhf`-architecture `mst0X` node. [ ]

Fix a number of typos. [ ][ ]

Update documentation. [ ][ ]

Fix Munin monitoring configuration for some nodes. [ ]

Fix the static IP address for a node. [ ]

In addition, Vagrant Cascadian updated host keys for the `cbxi4pro0` and `wbq0` nodes [ ] and, finally, node maintenance was also performed by Mattia Rizzolo [ ] and Holger Levsen [ ][ ][ ].

Contact As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

13 July 2022

Reproducible Builds: Reproducible Builds in June 2022

Welcome to the June 2022 report from the Reproducible Builds project. In these reports, we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.

Save the date! Despite several delays, we are pleased to announce dates for our in-person summit this year: November 1st 2022 November 3rd 2022
The event will happen in/around Venice (Italy), and we intend to pick a venue reachable via the train station and an international airport. However, the precise venue will depend on the number of attendees. Please see the announcement mail from Mattia Rizzolo, and do keep an eye on the mailing list for further announcements as it will hopefully include registration instructions.

News David Wheeler filed an issue against the Rust programming language to report that builds are not reproducible because full path to the source code is in the panic and debug strings . Luckily, as one of the responses mentions: the `--remap-path-prefix` solves this problem and has been used to great effect in build systems that rely on reproducibility (Bazel, Nix) to work at all and that there are efforts to teach cargo about it here .
The Python Security team announced that:
The `ctx` hosted project on PyPI was taken over via user account compromise and replaced with a malicious project which contained runtime code which collected the content of `os.environ.items()` when instantiating Ctx objects. The captured environment variables were sent as a base64 encoded query parameter to a Heroku application [ ]
As their announcement later goes onto state, version-pinning using hash-checking mode can prevent this attack, although this does depend on specific installations using this mode, rather than a prevention that can be applied systematically.
Developer vanitasvitae published an interesting and entertaining blog post detailing the blow-by-blow steps of debugging a reproducibility issue in PGPainless, a library which aims to make using OpenPGP in Java projects as simple as possible . Whilst their in-depth research into the internals of the `.jar` may have been unnecessary given that diffoscope would have identified the, it must be said that there is something to be said with occasionally delving into seemingly low-level details, as well describing any debugging process. Indeed, as vanitasvitae writes:
Yes, this would have spared me from 3h of debugging But I probably would also not have gone onto this little dive into the JAR/ZIP format, so in the end I m not mad.

Kees Cook published a short and practical blog post detailing how he uses reproducibility properties to aid work to replace one-element arrays in the Linux kernel. Kees approach is based on the principle that if a (small) proposed change is considered equivalent by the compiler, then the generated output will be identical but only if no other arbitrary or unrelated changes are introduced. Kees mentions the fantastic diffoscope tool, as well as various kernel-specific build options (eg. `KBUILD_BUILD_TIMESTAMP`) in order to prepare my build with the known to disrupt code layout options disabled .
Stefano Zacchiroli gave a presentation at GDR S curit Informatique based in part on a paper co-written with Chris Lamb titled Increasing the Integrity of Software Supply Chains. (Tweet)

Debian In Debian in this month, 28 reviews of Debian packages were added, 35 were updated and 27 were removed this month adding to our knowledge about identified issues. Two issue types were added: `nondeterministic_checksum_generated_by_coq` and `nondetermistic_js_output_from_webpack`. After Holger Levsen found hundreds of packages in the bookworm distribution that lack `.buildinfo` files, he uploaded 404 source packages to the archive (with no meaningful source changes). Currently bookworm now shows only 8 packages without `.buildinfo` files, and those 8 are fixed in unstable and should migrate shortly. By contrast, Debian unstable will always have packages without `.buildinfo` files, as this is how they come through the NEW queue. However, as these packages were not built on the official build servers (ie. they were uploaded by the maintainer) they will never migrate to Debian testing. In the future, therefore, testing should never have packages without `.buildinfo` files again. Roland Clobus posted yet another in-depth status report about his progress making the Debian Live images build reproducibly to our mailing list. In this update, Roland mentions that all major desktops build reproducibly with bullseye, bookworm and sid but also goes on to outline the progress made with automated testing of the generated images using openQA.

GNU Guix Vagrant Cascadian made a significant number of contributions to GNU Guix:

Submitted patches to fix reproducibility issues in keyutils and isl as well as reported two bugs affecting reproducibility testing [ ][ ].

23 specific fixes related to reproducibility. [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23]

Proposed setting `FORCE_SOURCE_DATE=1` in the environment of all builds in order to fix numerous timestamp issues in documentation generation tools.

Identified reproducibility issues in the `maradns` package as it appears to embed a random prime number. (Patch)

Responded in a thread to point out that GNU Guix already has the infrastructure in place to verify the reproducibility of downloaded substitutes for the vast majority of packages.

Lastly, Vagrant performed an evaluation of the unreproducible packages that remain in the distribution.

Elsewhere in GNU Guix, Ludovic Court s published a paper in the journal The Art, Science, and Engineering of Programming called Building a Secure Software Supply Chain with GNU Guix:
This paper focuses on one research question: how can [Guix]((https://www.gnu.org/software/guix/) and similar systems allow users to securely update their software? [ ] Our main contribution is a model and tool to authenticate new Git revisions. We further show how, building on Git semantics, we build protections against downgrade attacks and related threats. We explain implementation choices. This work has been deployed in production two years ago, giving us insight on its actual use at scale every day. The Git checkout authentication at its core is applicable beyond the specific use case of Guix, and we think it could benefit to developer teams that use Git.
A full PDF of the text is available.

openSUSE In the world of openSUSE, SUSE announced at SUSECon that they are preparing to meet SLSA level 4. (SLSA (Supply chain Levels for Software Artifacts) is a new industry-led standardisation effort that aims to protect the integrity of the software supply chain.) However, at the time of writing, timestamps within RPM archives are not normalised, so bit-for-bit identical reproducible builds are not possible. Some in-toto provenance files published for SUSE s SLE-15-SP4 as one result of the SLSA level 4 effort. Old binaries are not rebuilt, so only new builds (e.g. maintenance updates) have this metadata added. Lastly, Bernhard M. Wiedemann posted his usual monthly openSUSE reproducible builds status report.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `215`, `216` and `217` to Debian unstable. Chris Lamb also made the following changes:

New features:

Print profile output if we were called with `--profile` and we were killed via a `TERM` signal. This should help in situations where diffoscope is terminated due to some sort of timeout. [ ]

Support both PyPDF 1.x and 2.x. [ ]

Bug fixes:

Also catch `IndexError` exceptions (in addition to `ValueError`) when parsing `.pyc` files. (#1012258)

Correct the logic for supporting different versions of the `argcomplete` module. [ ]

Output improvements:

Don t leak the (likely-temporary) pathname when comparing PDF documents. [ ]

Logging improvements:

Update test fixtures for GNU readelf 2.38 (now in Debian unstable). [ ][ ]

Be more specific about the minimum required version of `readelf` (ie. binutils), as it appears that this patch level version change resulted in a change of output, not the minor version. [ ]

Use our `@skip_unless_tool_is_at_least` decorator (NB. `at_least`) over `@skip_if_tool_version_is` (NB. `is`) to fix tests under Debian stable. [ ]

Emit a warning if/when we are handling a UNIX `TERM` signal. [ ]

Codebase improvements:

Clarify in what situations the main `finally` block gets called with respect to `TERM` signal handling. [ ]

Clarify control flow in the `diffoscope.profiling` module. [ ]

Correctly package the `scripts/` directory. [ ]

In addition, Edward Betts updated a broken link to the RSS on the diffoscope homepage and Vagrant Cascadian updated the diffoscope package in GNU Guix [ ][ ][ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`build-compare` caused a regression for a few days.

`python-fasttext` (CPU-related issue).

Chris Lamb:

#1012614 filed against `node-dommatrix`.

#1012766 filed against `rtpengine`.

#1012790 filed against `sphinxcontrib-mermaid`.

#1012792 filed against `yaru-theme`.

#1012836 filed against `mapproxy` (forwarded upstream).

#1013257 filed against `libxsmm`.

#1014041 filed against `yt-dlp` (forwarded upstream).

#891263 was filed against puppet in February 2018 and the patch was finally proposed for inclusion upstream.

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

Holger Levsen:

Add a package set for packages that use the R programming language [ ] as well as one for Rust [ ].

Improve package set matching for Python [ ] and font-related [ ] packages.

Install the `lz4`, `lzop` and `xz-utils` packages on all nodes in order to detect running kernels. [ ]

Improve the cleanup mechanisms when testing the reproducibility of Debian Live images. [ ][ ]

In the automated node health checks, deprioritise the generic kernel warning . [ ]

Roland Clobus (Debian Live image reproducibility):

Add various maintenance jobs to the Jenkins view. [ ]

Cleanup old workspaces after 24 hours. [ ]

Cleanup temporary workspace and resulting directories. [ ]

Implement a number of fixes and improvements around publishing files. [ ][ ][ ]

Don t attempt to preserve the file timestamps when copying artifacts. [ ]

And finally, node maintenance was also performed by Mattia Rizzolo [ ].

Mailing list and website On our mailing list this month:

David Wheeler started a thread stating his desire that reproducible builds and GitBOM are able to work together simultaneously. David first describes the goals of both GitBOM and reproducibility, outlines the potential problems and even outlines a number of prospective solutions.

In a similar vein, David Wheeler also posted about the problems with Profile-Guided Optimisation (PGO) in relation to reproducible builds.

Roland Clobus copied in our mailing list with a question about whether enabling link-time optimisations (LTO) in Debian as a whole might cause reproducibility problems.

Mattia Rizzolo posted a request for assistance regarding the translations of our website.

Lastly, Chris Lamb updated the main Reproducible Builds website and documentation in a number of small ways, but primarily published an interview with Hans-Christoph Steiner of the F-Droid project. Chris Lamb also added a Coffeescript example for parsing and using the `SOURCE_DATE_EPOCH` environment variable [ ]. In addition, Sebastian Crane very-helpfully updated the screenshot of salsa.debian.org s request access button on the How to join the Salsa group. [ ]

Contact If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

1 July 2022

Paul Wise: FLOSS Activities June 2022

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

duck: add more obsolete domains (1 2), indicator phrases (1 2)

lintian: add more obsolete domains

Debian BTS usertags: fix old release goal tags, reassign edos tags to debian-qa, reassign sphinx tag to debian-python

Debian userdir-ldap: modernise and cleanup welcome message wording

Debian website: consultants fix

Debian package uploads: sptag

Debian wiki pages: AutomaticPackagingTools, Avahi, DebianJr, DebianPorts, Derivatives/Census, DeveloperNews, Firmware, Firmware/Open, Hardware/Database, PortsDocs/New, Ports (loongarch64, riscv64 (1, 2)), PortTemplate, RISC-V/32, SystemPrinting, Teams (DebianRakudoGroup, Lintian (1 2), pkg-fonts)

FOSSjobs wiki pages: Resources

Issues

New contributors needed for lintian

Conffile removal needed in dkms

Features in reportbug

Deprecated module usage in SPTAG

Embedded deps in SPTAG

Review

Spam: reported 5 Debian bug reports and 45 Debian mailing list posts

Debian wiki: RecentChanges for the month

Debian BTS usertags: changes for the month

Debian screenshots:

approved pforth timeshift virtualjaguar

rejected gcc (blank), gcc (GitHub repo card template), vagrant (low quality)

Administration

Debian wiki: unblock IP addresses, assist with account recovery, approve accounts

Communication

Initiate discussion about conveying Debian package porting details

Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors The sptag work was sponsored. All other work was done on a volunteer basis.

6 June 2022

Reproducible Builds: Reproducible Builds in May 2022

Welcome to the May 2022 report from the Reproducible Builds project. In our reports we outline the most important things that we have been up to over the past month. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Repfix paper Zhilei Ren, Shiwei Sun, Jifeng Xuan, Xiaochen Li, Zhide Zhou and He Jiang have published an academic paper titled Automated Patching for Unreproducible Builds:
[..] fixing unreproducible build issues poses a set of challenges [..], among which we consider the localization granularity and the historical knowledge utilization as the most significant ones. To tackle these challenges, we propose a novel approach [called] RepFix that combines tracing-based fine-grained localization with history-based patch generation mechanisms.
The paper (PDF, 3.5MB) uses the Debian `mylvmbackup` package as an example to show how RepFix can automatically generate patches to make software build reproducibly. As it happens, Reiner Herrmann submitted a patch for the `mylvmbackup` package which has remained unapplied by the Debian package maintainer for over seven years, thus this paper inadvertently underscores that achieving reproducible builds will require both technical and social solutions.

Python variables Johannes Schauer discovered a fascinating bug where simply naming your Python variable _m led to unreproducible .pyc files. In particular, the types module in Python 3.10 requires the following patch to make it reproducible:

--- a/Lib/types.py
+++ b/Lib/types.py
@@ -37,8 +37,8 @@ _ag = _ag()
 AsyncGeneratorType = type(_ag)
 
 class _C:
-    def _m(self): pass
-MethodType = type(_C()._m)
+    def _b(self): pass
+MethodType = type(_C()._b)

Simply renaming the dummy method from _m to _b was enough to workaround the problem. Johannes bug report first led to a number of improvements in diffoscope to aid in dissecting .pyc files, but upstream identified this as caused by an issue surrounding interned strings and is being tracked in CPython bug #78274.

New SPDX team to incorporate build metadata in Software Bill of Materials SPDX, the open standard for Software Bill of Materials (SBOM), is continuously developed by a number of teams and committees. However, SPDX has welcomed a new addition; a team dedicated to enhancing metadata about software builds, complementing reproducible builds in creating a more secure software supply chain. The SPDX Builds Team has been working throughout May to define the universal primitives shared by all build systems, including the who, what, where and how of builds:

Who: the identity of the person or organisation that controls the build infrastructure.

What: the inputs and outputs of a given build, combining metadata about the build s configuration with an SBOM describing source code and dependencies.

Where: the software packages making up the build system, from build orchestration tools such as Woodpecker CI and Tekton to language-specific tools.

How: the invocation of a build, linking metadata of a build to the identity of the person or automation tool that initiated it.

The SPDX Builds Team expects to have a usable data model by September, ready for inclusion in the SPDX 3.0 standard. The team welcomes new contributors, inviting those interested in joining to introduce themselves on the SPDX-Tech mailing list.

Talks at Debian Reunion Hamburg Some of the Reproducible Builds team (Holger Levsen, Mattia Rizzolo, Roland Clobus, Philip Rinn, etc.) met in real life at the Debian Reunion Hamburg (official homepage). There were several informal discussions amongst them, as well as two talks related to reproducible builds. First, Holger Levsen gave a talk on the status of Reproducible Builds for bullseye and bookworm and beyond (WebM, 210MB): Secondly, Roland Clobus gave a talk called Reproducible builds as applied to non-compiler output (WebM, 115MB):

Supply-chain security attacks This was another bumper month for supply-chain attacks in package repositories. Early in the month, Lance R. Vick noticed that the maintainer of the NPM `foreach` package let their personal email domain expire, so they bought it and now controls `foreach` on NPM and the 36,826 projects that depend on it . Shortly afterwards, Drew DeVault published a related blog post titled When will we learn? that offers a brief timeline of major incidents in this area and, not uncontroversially, suggests that the correct way to ship packages is with your distribution s package manager .

Bootstrapping Bootstrapping is a process for building software tools progressively from a primitive compiler tool and source language up to a full Linux development environment with GCC, etc. This is important given the amount of trust we put in existing compiler binaries. This month, a bootstrappable mini-kernel was announced. Called boot2now, it comprises a series of compilers in the form of bootable machine images.

Google s new Assured Open Source Software service Google Cloud (the division responsible for the Google Compute Engine) announced a new Assured Open Source Software service. Noting the considerable 650% year-over-year increase in cyberattacks aimed at open source suppliers, the new service claims to enable enterprise and public sector users of open source software to easily incorporate the same OSS packages that Google uses into their own developer workflows . The announcement goes on to enumerate that packages curated by the new service would be:

Regularly scanned, analyzed, and fuzz-tested for vulnerabilities.

Have corresponding enriched metadata incorporating Container/Artifact Analysis data.

Are built with Cloud Build including evidence of verifiable SLSA-compliance

Are verifiably signed by Google.

Are distributed from an Artifact Registry secured and protected by Google.

(Full announcement)

A retrospective on the Rust programming language Andrew bunnie Huang published a long blog post this month promising a critical retrospective on the Rust programming language. Amongst many acute observations about the evolution of the language s syntax (etc.), the post beings to critique the languages approach to supply chain security ( Rust Has A Limited View of Supply Chain Security ) and reproducibility ( You Can t Reproduce Someone Else s Rust Build ):
There s some bugs open with the Rust maintainers to address reproducible builds, but with the number of issues they have to deal with in the language, I am not optimistic that this problem will be resolved anytime soon. Assuming the only driver of the unreproducibility is the inclusion of OS paths in the binary, one fix to this would be to re-configure our build system to run in some sort of a chroot environment or a virtual machine that fixes the paths in a way that almost anyone else could reproduce. I say almost anyone else because this fix would be OS-dependent, so we d be able to get reproducible builds under, for example, Linux, but it would not help Windows users where chroot environments are not a thing.
(Full post)

Reproducible Builds IRC meeting The minutes and logs from our May 2022 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on Tuesday 28th June at 15:00 UTC on `#reproducible-builds` on the OFTC network.

A new tool to improve supply-chain security in Arch Linux kpcyrd published yet another interesting tool related to reproducibility. Writing about the tool in a recent blog post, kpcyrd mentions that although many `PKGBUILDs` provide authentication in the context of signed Git tags (i.e. the ability to verify the Git tag was signed by one of the two trusted keys ), they do not support pinning, ie. that upstream could create a new signed Git tag with an identical name, and arbitrarily change the source code without the [maintainer] noticing . Conversely, other `PKGBUILD`s support pinning but not authentication. The new tool, auth-tarball-from-git, fixes both problems, as nearly outlined in kpcyrd s original blog post.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `212`, `213` and `214` to Debian unstable. Chris also made the following changes:

New features:

Add support for extracting `vmlinuz` Linux kernel images. [ ]

Support both `python-argcomplete` 1.x and 2.x. [ ]

Strip `sticky` etc. from `x.deb: sticky Debian binary package [ ]`. [ ]

Integrate test coverage with GitLab s concept of artifacts. [ ][ ][ ]

Bug fixes:

Don t mask differences in `.zip` or `.jar` central directory extra fields. [ ]

Don t show a binary comparison of `.zip` or `.jar` files if we have observed at least one nested difference. [ ]

Codebase improvements:

Substantially update comment for our calls to `zipinfo` and `zipinfo -v`. [ ]

Use `assert_diff` in `test_zip` over calling `get_data` with a separate `assert`. [ ]

Don t call `re.compile` and then call `.sub` on the result; just call `re.sub` directly. [ ]

Clarify the comment around the difference between `--usage` and `--help`. [ ]

Testsuite improvements:

Test `--help` and `--usage`. [ ]

Test that `--help` includes the file formats. [ ]

Vagrant Cascadian added an external tool reference `xb-tool` for GNU Guix [ ] as well as updated the diffoscope package in GNU Guix itself [ ][ ][ ].

Distribution work In Debian, 41 reviews of Debian packages were added, 85 were updated and 13 were removed this month adding to our knowledge about identified issues. A number of issue types have been updated, including adding a new `nondeterministic_ordering_in_deprecated_items_collected_by_doxygen` toolchain issue [ ] as well as ones for `mono_mastersummary_xml_files_inherit_filesystem_ordering` [ ], `extended_attributes_in_jar_file_created_without_manifest` [ ] and `apxs_captures_build_path` [ ]. Vagrant Cascadian performed a rough check of the reproducibility of core package sets in GNU Guix, and in openSUSE, Bernhard M. Wiedemann posted his usual monthly reproducible builds status report.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`gtkmm-documentation` (merged; sorting issue)

`librespot` (merged; random `BuildID` issue)

`lirc` (merged)

`lsof` (`uname`/`hostname` problem)

`solanum` (merged, possibly a race condition)

Chris Lamb:

#1010845 filed against `logapp`.

#1010854 filed against `mono`.

#1010855 filed against `longrun`.

#1011452 filed against `rust-simplelog`.

#1011752 filed against `freesas`.

`PeachPy` (merged, sort entries)

Johannes Schauer Marin Rodrigues:

#1010957 filed against `man-db`.

Vagrant Cascadian:

#1010462 filed against `mtink`.

#1010463 filed against `fceux`.

#1010466 filed against `glob2`.

#1010483 filed against `coinor-cgl`.

#1010486 filed against `metapixel`.

#1010781 filed against `ragel`.

#1010785 filed against `gdome2`.

#1010787 filed against `sgml-base-doc`.

#1010789 filed against `xarclock`.

#1010790 filed against `xgammon`.

#1010825 filed against `lwatch`.

#1010828 filed against `bbrun`.

#1010830 filed against `gscanbus`.

#1010859 filed against `libnss-gw-name`.

#1010870 filed against `pidgin-blinklight`.

#1010871 filed against `dvbtune`.

#1010872 filed against `efax`.

#1010944 filed against `quelcom`.

#1010948 filed against `xine-lib-1.2`.

#1011034 filed against `fusesmb`.

#1011036 filed against `mailfront`.

#1011104 filed against `convlit`.

#1011109 filed against `bitstormlite`.

#1011257 filed against `coinor-osi`.

#1011402 filed against `razor`.

#1011405 filed against `autoclass`.

#1011428 filed against `cdbackup`.

#1011429 filed against `dds2tar`.

#1011469 filed against `transcalc`.

#1011470 filed against `libapache2-mod-authz-unixgroup`.

#1011471 filed against `mgdiff`.

#1011478 filed against `scsitools`.

#1011479 filed against `fstrcmp`.

#1011480 filed against `libxsettings-client`.

#1011481 filed against `tamil-gtk2im`.

#1011486 filed against `tdfsb`.

#1011488 filed against `stymulator`.

#1011489 filed against `wiipdf`.

#1011490 filed against `gdigi`.

#1011491 filed against `getstream`.

#1011493 filed against `freecdb`.

#1011495 filed against `modglue`.

#1011496 filed against `nwall`.

#1011498 filed against `parprouted`.

#1011499 filed against `imagination`.

#1011500 filed against `tuxcmd-modules`.

#1011501 filed against `libapache2-mod-authn-yubikey`.

#1011503 filed against `libapache2-mod-auth-plain`.

lcsync (remove build paths)

Reproducible builds website Chris Lamb updated the main Reproducible Builds website and documentation in a number of small ways, but also prepared and published an interview with Jan Nieuwenhuizen about Bootstrappable Builds, GNU Mes and GNU Guix. [ ][ ][ ][ ] In addition, Tim Jones added a link to the Talos Linux project [ ] and billchenchina fixed a dead link [ ].

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

Holger Levsen:

Add support for detecting running kernels that require attention. [ ]

Temporarily configure a host to support performing Debian builds for packages that lack `.buildinfo` files. [ ]

Update generated webpages to clarify wishes for feedback. [ ]

Update copyright years on various scripts. [ ]

Mattia Rizzolo:

Provide a facility so that Debian Live image generation can copy a file remotely. [ ][ ][ ][ ]

Roland Clobus:

Add initial support for testing generated images with OpenQA. [ ]

And finally, as usual, node maintenance was also performed by Holger Levsen [ ][ ].

Misc news On our mailing list this month:

John Neffenger posted that the early-access release of OpenJDK version 19 build 21 is reproducible.

Mattia Rizzolo added a request around tentatively planning a Reproducible Builds summit in 2022.

Bernhard M. Wiedemann posted about a Reproducible Builds meetup at the openSUSE conference in Nuremberg.

Luca Boccassi asked for help getting `arm64` binaries to build reproducibly for the Debian `systemd` package.

Yaobin Wen asked a number of questions in an attempt to discover the best practices for manage Debian `.dsc` files using reprepro.

Contact If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

27 May 2022

Reproducible Builds (diffoscope): diffoscope 214 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 214. This version includes the following changes:

[ Chris Lamb ]
* Support both python-argcomplete 1.x and 2.x.
[ Vagrant Cascadian ]
* Add external tool on GNU Guix for xb-tool.

You find out more by visiting the project homepage.

5 May 2022

Reproducible Builds: Reproducible Builds in April 2022

Welcome to the April 2022 report from the Reproducible Builds project! In these reports, we try to summarise the most important things that we have been up to over the past month. If you are interested in contributing to the project, please take a few moments to visit our Contribute page on our website.

News Cory Doctorow published an interesting article this month about the possibility of Undetectable backdoors for machine learning models. Given that machine learning models can provide unpredictably incorrect results, Doctorow recounts that there exists another category of adversarial examples that comprise a gimmicked machine-learning input that, to the human eye, seems totally normal but which causes the ML system to misfire dramatically that permit the possibility of planting undetectable back doors into any machine learning system at training time .
Chris Lamb published two supporter spotlights on our blog: the first about Amateur Radio Digital Communications (ARDC) and the second about the Google Open Source Security Team (GOSST).
Piergiorgio Ladisa, Henrik Plate, Matias Martinez and Olivier Barais published a new academic paper titled A Taxonomy of Attacks on Open-Source Software Supply Chains (PDF):
This work proposes a general taxonomy for attacks on open-source supply chains, independent of specific programming languages or ecosystems, and covering all supply chain stages from code contributions to package distribution. Taking the form of an attack tree, it covers 107 unique vectors, linked to 94 real-world incidents, and mapped to 33 mitigating safeguards.

Elsewhere in academia, Ly Vu Duc published his PhD thesis. Titled Towards Understanding and Securing the OSS Supply Chain (PDF), Duc s abstract reads as follows:
This dissertation starts from the first link in the software supply chain, developers . Since many developers do not update their vulnerable software libraries, thus exposing the user of their code to security risks. To understand how they choose, manage and update the libraries, packages, and other Open-Source Software (OSS) that become the building blocks of companies completed products consumed by end-users, twenty-five semi-structured interviews were conducted with developers of both large and small-medium enterprises in nine countries. All interviews were transcribed, coded, and analyzed according to applied thematic analysis

Upstream news Filippo Valsorda published an informative blog post recently called How Go Mitigates Supply Chain Attacks outlining the high-level features of the Go ecosystem that helps prevent various supply-chain attacks.
There was new/further activity on a pull request filed against openssl by Sebastian Andrzej Siewior in order to prevent saved `CFLAGS` (which may contain the `-fdebug-prefix-map=<PATH>` flag that is used to strip an arbitrary the build path from the debug info if this information remains recorded then the binary is no longer reproducible if the build directory changes.

Events The Linux Foundation s SupplyChainSecurityCon, will take place June 21st 24th 2022, both virtually and in Austin, Texas. Long-time Reproducible Builds and openSUSE contributor Bernhard M. Wiedemann learned that he had his talk accepted, and will speak on Reproducible Builds: Unexpected Benefits and Problems on June 21st.
There will be an in-person Debian Reunion in Hamburg, Germany later this year, taking place from 23 30 May. Although this is a Debian event, there will be some folks from the broader Reproducible Builds community and, of course, everyone is welcome. Please see the event page on the Debian wiki for more information. 41 people have registered so far, and there s approx 10 on-site beds still left.
The minutes and logs from our April 2022 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on May 31st at 15:00 UTC on `#reproducible-builds` on the OFTC network.

Debian Roland Clobus wrote another in-depth status update about the status of live Debian images, summarising the current situation that all major desktops build reproducibly with bullseye, bookworm and sid, including the Cinnamon desktop on bookworm and sid, but at a small functionality cost: 14 words will be incorrectly abbreviated . This work incorporated:

Reporting an issue about unnecessarily modified timestamps in the daily Debian installer images. [ ]

Reporting a bug against the `debian-installer`: in order to use a suitable kernel version. (#1006800)

Reporting a bug in: `texlive-binaries` regarding the unreproducible content of `.fmt` files. (#1009196)

Adding hacks to make the Cinnamon desktop image reproducible in bookworm and sid. [ ]

Added a script to rebuild a live-build ISO image from a given timestamp. [

etc.

On our mailing list, Venkata Pyla started a thread on the Debian debconf cache is non-reproducible issue while creating system images and Vagrant Cascadian posted an excellent summary of the reproducibility status of core package sets in Debian and solicited for similar information from other distributions.
Lastly, 122 reviews of Debian packages were added, 44 were updated and 193 were removed this month adding to our extensive knowledge about identified issues. A number of issue types have been updated as well, including `timestamps_generated_by_hevea`, `randomness_in_ocaml_preprocessed_files`, `build_path_captured_in_emacs_el_file`, `golang_compiler_captures_build_path_in_binary` and `build_path_captured_in_assembly_objects`,

Other distributions Happy birthday to GNU Guix, which recently turned 10 years old! People have been sharing their stories, in which reproducible builds and bootstrappable builds are a recurring theme as a feature important to its users and developers. The experiences are available on the GNU Guix blog as well as a post on fossandcrafts.org
In openSUSE, Bernhard M. Wiedemann posted his usual monthly reproducible builds status report.

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:

`python-numcodecs` (CPU-related issue)

`salt` (date issue)

`golang-github-prometheus-node_exporter` (date-related issue)

`librsb` (report compile-time CPU-detection)

Chris Lamb:

#1009981 filed against `rsync`.

#1010279 filed against `python-iso8601`.

#1010316 filed against `python-cai`.

#1010318 filed against `datalad`.

Johannes Schauer Marin Rodrigues:

#1010368 filed against `python3.10`.

Roland Clobus:

#1009196 filed against `texlive-bin`.

Vagrant Cascadian:

#1009337 filed against `python-public`.

#1009339 filed against `python-gitlab`.

#1009342 filed against `xfce4-panel-profiles`.

#1009796 & #1009797 filed against `apt`.

#1009799 filed against `puppet`.

#1009931 filed against `gmp`.

#1009934 filed against `openssl`.

#1010043 filed against `serf`.

#1010233 filed against `glibc`.

#1010238 filed against `binutils`.

#1010378 filed against `leds-alix`.

#1010379 filed against `snacc`.

#1010414 filed against `libcec`.

diffoscope diffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb prepared and uploaded versions `210` and `211` to Debian unstable, as well as noticed that some Python `.pyc` files are reported as `data`, so we should support `.pyc` as a fallback filename extension [ ]. In addition, Mattia Rizzolo disabled the Gnumeric tests in Debian as the package is not currently available [ ] and dropped mplayer from `Build-Depends` too [ ]. In addition, Mattia fixed an issue to ensure that the `PATH` environment variable is properly modified for all actions, not just when running the comparator. [ ]

Testing framework The Reproducible Builds project runs a significant testing framework at tests.reproducible-builds.org, to check packages and other artifacts for reproducibility. This month, the following changes were made:

Daniel Golle:

Prefer a different solution to avoid building all OpenWrt packages; skip packages from optional community feeds. [ ]

Holger Levsen:

Detect Python deprecation warnings in the node health check. [ ]

Detect failure to build the Debian Installer. [ ]

Mattia Rizzolo:

Install disorderfs for building OpenWrt packages. [ ]

Paul Spooren (OpenWrt-related changes):

Don t build all packages whilst the core packages are not yet reproducible. [ ]

Add a missing `RUN` directive to `node_cleanup`. [ ]

Be less verbose during a toolchain build. [ ]

Use disorderfs for rebuilds and update the documentation to match. [ ][ ][ ]

Roland Clobus:

Publish the last reproducible Debian ISO image. [ ]

Use the `rebuild.sh` script from the `live-build` package. [ ]

Lastly, node maintenance was also performed by Holger Levsen [ ][ ].
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: `#reproducible-builds` on `irc.oftc.net`.

Twitter: @ReproBuilds

Mailing list: `rb-general@lists.reproducible-builds.org`

27 April 2022

Antoine Beaupr : building Debian packages under qemu with sbuild

I've been using sbuild for a while to build my Debian packages, mainly because it's what is used by the Debian autobuilders, but also because it's pretty powerful and efficient. Configuring it just right, however, can be a challenge. In my quick Debian development guide, I had a few pointers on how to configure sbuild with the normal schroot setup, but today I finished a qemu based configuration.

Why I want to use qemu mainly because it provides better isolation than a chroot. I sponsor packages sometimes and while I typically audit the source code before building, it still feels like the extra protection shouldn't hurt. I also like the idea of unifying my existing virtual machine setup with my build setup. My current VM is kind of all over the place: libvirt, vagrant, GNOME Boxes, etc?). I've been slowly converging over libvirt however, and most solutions I use right now rely on qemu under the hood, certainly not chroots... I could also have decided to go with containers like LXC, LXD, Docker (with conbuilder, whalebuilder, docker-buildpackage), systemd-nspawn (with debspawn), unshare (with `schroot --chroot-mode=unshare`), or whatever: I didn't feel those offer the level of isolation that is provided by qemu. The main downside of this approach is that it is (obviously) slower than native builds. But on modern hardware, that cost should be minimal.

How Basically, you need this:

sudo mkdir -p /srv/sbuild/qemu/
sudo apt install sbuild-qemu
sudo sbuild-qemu-create -o /srv/sbuild/qemu/unstable.img unstable https://deb.debian.org/debian

Then to make this used by default, add this to ~/.sbuildrc:

# run autopkgtest inside the schroot
$run_autopkgtest = 1;
# tell sbuild to use autopkgtest as a chroot
$chroot_mode = 'autopkgtest';
# tell autopkgtest to use qemu
$autopkgtest_virt_server = 'qemu';
# tell autopkgtest-virt-qemu the path to the image
# use --debug there to show what autopkgtest is doing
$autopkgtest_virt_server_options = [ '--', '/srv/sbuild/qemu/%r-%a.img' ];
# tell plain autopkgtest to use qemu, and the right image
$autopkgtest_opts = [ '--', 'qemu', '/srv/sbuild/qemu/%r-%a.img' ];
# no need to cleanup the chroot after build, we run in a completely clean VM
$purge_build_deps = 'never';
# no need for sudo
$autopkgtest_root_args = '';

Note that the above will use the default autopkgtest (1GB, one core) and qemu (128MB, one core) configuration, which might be a little low on resources. You probably want to be explicit about this, with something like this:

# extra parameters to pass to qemu
# --enable-kvm is not necessary, detected on the fly by autopkgtest
my @_qemu_options = ['--ram-size=4096', '--cpus=2'];
# tell autopkgtest-virt-qemu the path to the image
# use --debug there to show what autopkgtest is doing
$autopkgtest_virt_server_options = [ @_qemu_options, '--', '/srv/sbuild/qemu/%r-%a.img' ];
$autopkgtest_opts = [ '--', 'qemu', @qemu_options, '/srv/sbuild/qemu/%r-%a.img'];

This configuration will:

create a virtual machine image in /srv/sbuild/qemu for unstable
tell sbuild to use that image to create a temporary VM to build the packages
tell sbuild to run autopkgtest (which should really be default)
tell autopkgtest to use qemu for builds and for tests

Note that the VM created by sbuild-qemu-create have an unlocked root account with an empty password.

Other useful tasks

enter the VM to make test, changes will be discarded (thanks Nick Brown for the sbuild-qemu-boot tip!):

 sbuild-qemu-boot /srv/sbuild/qemu/unstable-amd64.img

That program is shipped only with bookworm and later, an equivalent command is:

 qemu-system-x86_64 -snapshot -enable-kvm -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -m 2048 -nographic /srv/sbuild/qemu/unstable-amd64.img

The key argument here is -snapshot.

enter the VM to make permanent changes, which will not be discarded:

 sudo sbuild-qemu-boot --readwrite /srv/sbuild/qemu/unstable-amd64.img

Equivalent command:

 sudo qemu-system-x86_64 -enable-kvm -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -m 2048 -nographic /srv/sbuild/qemu/unstable-amd64.img

update the VM (thanks lavamind):

 sudo sbuild-qemu-update /srv/sbuild/qemu/unstable-amd64.img

build in a specific VM regardless of the suite specified in the changelog (e.g. UNRELEASED, bookworm-backports, bookworm-security, etc):
```
 sbuild --autopkgtest-virt-server-opts="-- qemu /var/lib/sbuild/qemu/bookworm-amd64.img"
```
Note that you'd also need to pass --autopkgtest-opts if you want autopkgtest to run in the correct VM as well:
```
 sbuild --autopkgtest-opts="-- qemu /var/lib/sbuild/qemu/unstable.img" --autopkgtest-virt-server-opts="-- qemu /var/lib/sbuild/qemu/bookworm-amd64.img"
```
You might also need parameters like --ram-size if you customized it above.

And yes, this is all quite complicated and could be streamlined a little, but that's what you get when you have years of legacy and just want to get stuff done. It seems to me autopkgtest-virt-qemu should have a magic flag starts a shell for you, but it doesn't look like that's a thing. When that program starts, it just says ok and sits there. Maybe because the authors consider the above to be simple enough (see also bug #911977 for a discussion of this problem).

Live access to a running test When autopkgtest starts a VM, it uses this funky qemu commandline:

qemu-system-x86_64 -m 4096 -smp 2 -nographic -net nic,model=virtio -net user,hostfwd=tcp:127.0.0.1:10022-:22 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,id=rng-device0 -monitor unix:/tmp/autopkgtest-qemu.w1mlh54b/monitor,server,nowait -serial unix:/tmp/autopkgtest-qemu.w1mlh54b/ttyS0,server,nowait -serial unix:/tmp/autopkgtest-qemu.w1mlh54b/ttyS1,server,nowait -virtfs local,id=autopkgtest,path=/tmp/autopkgtest-qemu.w1mlh54b/shared,security_model=none,mount_tag=autopkgtest -drive index=0,file=/tmp/autopkgtest-qemu.w1mlh54b/overlay.img,cache=unsafe,if=virtio,discard=unmap,format=qcow2 -enable-kvm -cpu kvm64,+vmx,+lahf_lm

... which is a typical qemu commandline, I'm sorry to say. That gives us a VM with those settings (paths are relative to a temporary directory, /tmp/autopkgtest-qemu.w1mlh54b/ in the above example):

the shared/ directory is, well, shared with the VM
port 10022 is forward to the VM's port 22, presumably for SSH, but not SSH server is started by default
the ttyS1 and ttyS2 UNIX sockets are mapped to the first two serial ports (use nc -U to talk with those)
the monitor UNIX socket is a qemu control socket (see the QEMU monitor documentation, also nc -U)

In other words, it's possible to access the VM with:

nc -U /tmp/autopkgtest-qemu.w1mlh54b/ttyS2

The nc socket interface is ... not great, but it works well enough. And you can probably fire up an SSHd to get a better shell if you feel like it.

Nitty-gritty details no one cares about

Fixing hang in sbuild cleanup I'm having a hard time making heads or tails of this, but please bear with me. In sbuild + schroot, there's this notion that we don't really need to cleanup after ourselves inside the schroot, as the schroot will just be delted anyways. This behavior seems to be handled by the internal "Session Purged" parameter. At least in lib/Sbuild/Build.pm, we can see this:

my $is_cloned_session = (defined ($session->get('Session Purged')) &&
             $session->get('Session Purged') == 1) ? 1 : 0;
[...]
if ($is_cloned_session)  
$self->log("Not cleaning session: cloned chroot in use\n");
  else  
if ($purge_build_deps)  
    # Removing dependencies
    $resolver->uninstall_deps();
  else  
    $self->log("Not removing build depends: as requested\n");

The schroot builder defines that parameter as:

    $self->set('Session Purged', $info-> 'Session Purged' );

... which is ... a little confusing to me. $info is:

my $info = $self->get('Chroots')->get_info($schroot_session);

... so I presume that depends on whether the schroot was correctly cleaned up? I stopped digging there... ChrootUnshare.pm is way more explicit:

$self->set('Session Purged', 1);

I wonder if we should do something like this with the autopkgtest backend. I guess people might technically use it with something else than qemu, but qemu is the typical use case of the autopkgtest backend, in my experience. Or at least certainly with things that cleanup after themselves. Right? For some reason, before I added this line to my configuration:

$purge_build_deps = 'never';

... the "Cleanup" step would just completely hang. It was quite bizarre.

Disgression on the diversity of VM-like things There are a lot of different virtualization solutions one can use (e.g. Xen, KVM, Docker or Virtualbox). I have also found libguestfs to be useful to operate on virtual images in various ways. Libvirt and Vagrant are also useful wrappers on top of the above systems. There are particularly a lot of different tools which use Docker, Virtual machines or some sort of isolation stronger than chroot to build packages. Here are some of the alternatives I am aware of:

Whalebuilder - Docker builder

conbuilder - "container" builder

debspawn - system-nspawn builder

docker-buildpackage - Docker builder

qemubuilder - qemu builder

qemu-sbuild-utils - qemu + sbuild + autopkgtest

Take, for example, Whalebuilder, which uses Docker to build packages instead of `pbuilder` or `sbuild`. Docker provides more isolation than a simple `chroot`: in `whalebuilder`, packages are built without network access and inside a virtualized environment. Keep in mind there are limitations to Docker's security and that `pbuilder` and `sbuild` do build under a different user which will limit the security issues with building untrusted packages. On the upside, some of things are being fixed: `whalebuilder` is now an official Debian package (whalebuilder) and has added the feature of passing custom arguments to dpkg-buildpackage. None of those solutions (except the `autopkgtest`/`qemu` backend) are implemented as a sbuild plugin, which would greatly reduce their complexity. I was previously using Qemu directly to run virtual machines, and had to create VMs by hand with various tools. This didn't work so well so I switched to using Vagrant as a de-facto standard to build development environment machines, but I'm returning to Qemu because it uses a similar backend as KVM and can be used to host longer-running virtual machines through libvirt. The great thing now is that `autopkgtest` has good support for `qemu` and `sbuild` has bridged the gap and can use it as a build backend. I originally had found those bugs in that setup, but all of them are now fixed:

#911977: sbuild: how do we correctly guess the VM name in autopkgtest?

#911979: sbuild: fails on chown in autopkgtest-qemu backend

#911963: autopkgtest qemu build fails with proxy_cmd: parameter not set

#911981: autopkgtest: qemu server warns about missing CPU features

So we have unification! It's possible to run your virtual machines and Debian builds using a single VM image backend storage, which is no small feat, in my humble opinion. See the sbuild-qemu blog post for the annoucement Now I just need to figure out how to merge Vagrant, GNOME Boxes, and libvirt together, which should be a matter of placing images in the right place... right? See also hosting.

pbuilder vs sbuild I was previously using `pbuilder` and switched in 2017 to `sbuild`. AskUbuntu.com has a good comparative between pbuilder and sbuild that shows they are pretty similar. The big advantage of sbuild is that it is the tool in use on the buildds and it's written in Perl instead of shell. My concerns about switching were POLA (I'm used to pbuilder), the fact that pbuilder runs as a separate user (works with sbuild as well now, if the `_apt` user is present), and setting up COW semantics in sbuild (can't just plug cowbuilder there, need to configure overlayfs or aufs, which was non-trivial in Debian jessie). Ubuntu folks, again, have more documentation there. Debian also has extensive documentation, especially about how to configure overlays. I was ultimately convinced by stapelberg's post on the topic which shows how much simpler sbuild really is...

Who Thanks lavamind for the introduction to the `sbuild-qemu` package.

Next.

Previous.

Search Results: "vagrant"

6 April 2023

5 March 2023

6 February 2023

Distribution work

2 February 2023

20 January 2023

7 January 2023

8 December 2022

29 November 2022

11 November 2022

16 October 2022

7 October 2022

Other distributions In openSUSE, Bernhard M. Wiedemann published his usual openSUSE monthly report.

29 September 2022

9 September 2022

4 August 2022

13 July 2022

Contact If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via: IRC: #reproducible-builds on irc.oftc.net. Twitter: @ReproBuilds Mailing list: rb-general@lists.reproducible-builds.org

1 July 2022

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Issues New contributors needed for lintian Conffile removal needed in dkms Features in reportbug Deprecated module usage in SPTAG Embedded deps in SPTAG

Review Spam: reported 5 Debian bug reports and 45 Debian mailing list posts Debian wiki: RecentChanges for the month Debian BTS usertags: changes for the month Debian screenshots: approved pforth timeshift virtualjaguar rejected gcc (blank), gcc (GitHub repo card template), vagrant (low quality)

Administration Debian wiki: unblock IP addresses, assist with account recovery, approve accounts

Communication Initiate discussion about conveying Debian package porting details Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors The sptag work was sponsored. All other work was done on a volunteer basis.

6 June 2022

Reproducible Builds IRC meeting The minutes and logs from our May 2022 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on Tuesday 28th June at 15:00 UTC on #reproducible-builds on the OFTC network.

Contact If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via: IRC: #reproducible-builds on irc.oftc.net. Twitter: @ReproBuilds Mailing list: rb-general@lists.reproducible-builds.org

27 May 2022

5 May 2022

27 April 2022

Issues

New contributors needed for lintian

Conffile removal needed in dkms

Features in reportbug

Deprecated module usage in SPTAG

Embedded deps in SPTAG

Administration

Debian wiki: unblock IP addresses, assist with account recovery, approve accounts

Communication

Initiate discussion about conveying Debian package porting details

Respond to queries from Debian users and contributors on the mailing lists and IRC

Reproducible Builds IRC meeting The minutes and logs from our May 2022 IRC meeting have been published. In case you missed this one, our next IRC meeting will take place on Tuesday 28th June at 15:00 UTC on `#reproducible-builds` on the OFTC network.